Sep 18 13:58:46.644478 ip-10-0-143-120 systemd[1]: Starting Kubernetes Kubelet...
Sep 18 13:58:49.840894 ip-10-0-143-120 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 13:58:49.840894 ip-10-0-143-120 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 18 13:58:49.840894 ip-10-0-143-120 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 13:58:49.840894 ip-10-0-143-120 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 18 13:58:49.840894 ip-10-0-143-120 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 13:58:49.841722 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.840963    2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 18 13:58:49.844889 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.844871    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 13:58:49.844889 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.844888    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 13:58:49.844954 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.844892    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 13:58:49.844954 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.844895    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 13:58:49.844954 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.844898    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 13:58:49.844954 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.844901    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 13:58:49.844954 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.844905    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 13:58:49.844954 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.844908    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 13:58:49.844954 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.844911    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 13:58:49.844954 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.844914    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 13:58:49.844954 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.844916    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 13:58:49.844954 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.844924    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 13:58:49.844954 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.844927    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 13:58:49.844954 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.844930    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 13:58:49.844954 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.844933    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 13:58:49.844954 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.844935    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 13:58:49.844954 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.844938    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 13:58:49.844954 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.844941    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 13:58:49.844954 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.844944    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 13:58:49.844954 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.844947    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 13:58:49.844954 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.844949    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 13:58:49.844954 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.844952    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 13:58:49.845447 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.844955    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 13:58:49.845447 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.844957    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 13:58:49.845447 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.844960    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 13:58:49.845447 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.844963    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 13:58:49.845447 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.844966    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 13:58:49.845447 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.844970    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 13:58:49.845447 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.844973    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 13:58:49.845447 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.844975    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 13:58:49.845447 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.844978    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 13:58:49.845447 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.844981    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 13:58:49.845447 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.844983    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 13:58:49.845447 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.844986    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 13:58:49.845447 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.844988    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 13:58:49.845447 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.844992    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 13:58:49.845447 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.844994    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 13:58:49.845447 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.844998    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 13:58:49.845447 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845002    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 13:58:49.845447 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845005    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 13:58:49.845447 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845008    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 13:58:49.845447 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845010    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 13:58:49.846032 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845014    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 13:58:49.846032 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845016    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 13:58:49.846032 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845019    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 13:58:49.846032 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845022    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 13:58:49.846032 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845024    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 13:58:49.846032 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845027    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 13:58:49.846032 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845029    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 13:58:49.846032 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845032    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 13:58:49.846032 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845034    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 13:58:49.846032 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845037    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 13:58:49.846032 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845039    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 13:58:49.846032 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845042    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 13:58:49.846032 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845044    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 13:58:49.846032 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845047    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 13:58:49.846032 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845050    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 13:58:49.846032 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845053    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 13:58:49.846032 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845055    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 13:58:49.846032 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845059    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 13:58:49.846032 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845063    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 13:58:49.846518 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845066    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 13:58:49.846518 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845068    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 13:58:49.846518 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845071    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 13:58:49.846518 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845074    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 13:58:49.846518 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845076    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 13:58:49.846518 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845079    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 13:58:49.846518 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845081    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 13:58:49.846518 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845084    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 13:58:49.846518 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845087    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 13:58:49.846518 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845091    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 13:58:49.846518 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845095    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 13:58:49.846518 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845098    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 13:58:49.846518 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845100    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 13:58:49.846518 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845103    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 13:58:49.846518 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845107    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 13:58:49.846518 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845109    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 13:58:49.846518 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845112    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 13:58:49.846518 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845114    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 13:58:49.846518 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845117    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 13:58:49.846518 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845119    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 13:58:49.847018 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845122    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 13:58:49.847018 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845125    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 13:58:49.847018 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845127    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 13:58:49.847018 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845130    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 13:58:49.847018 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845132    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 13:58:49.847018 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845135    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 13:58:49.847018 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845139    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 13:58:49.847018 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845142    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 13:58:49.847018 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845145    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 13:58:49.847018 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845147    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 13:58:49.847018 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845150    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 13:58:49.847018 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845152    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 13:58:49.847018 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845154    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 13:58:49.847018 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845157    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 13:58:49.847018 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845160    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 13:58:49.847018 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845162    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 13:58:49.847018 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845165    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 13:58:49.847018 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845181    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 13:58:49.847018 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845879    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 13:58:49.847018 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845885    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 13:58:49.847525 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845889    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 13:58:49.847525 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845893    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 13:58:49.847525 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845896    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 13:58:49.847525 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845899    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 13:58:49.847525 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845902    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 13:58:49.847525 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845905    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 13:58:49.847525 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845907    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 13:58:49.847525 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845910    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 13:58:49.847525 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845913    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 13:58:49.847525 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845922    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 13:58:49.847525 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845925    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 13:58:49.847525 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845928    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 13:58:49.847525 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845930    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 13:58:49.847525 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845933    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 13:58:49.847525 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845936    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 13:58:49.847525 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845938    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 13:58:49.847525 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845941    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 13:58:49.847525 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845944    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 13:58:49.847525 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845947    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 13:58:49.847525 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845950    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 13:58:49.848040 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845953    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 13:58:49.848040 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845956    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 13:58:49.848040 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845959    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 13:58:49.848040 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845961    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 13:58:49.848040 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845964    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 13:58:49.848040 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845966    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 13:58:49.848040 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845969    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 13:58:49.848040 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845972    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 13:58:49.848040 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845974    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 13:58:49.848040 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845977    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 13:58:49.848040 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845979    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 13:58:49.848040 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845982    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 13:58:49.848040 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845984    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 13:58:49.848040 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845987    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 13:58:49.848040 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845989    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 13:58:49.848040 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.845992    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 13:58:49.848040 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846001    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 13:58:49.848040 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846003    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 13:58:49.848040 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846006    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 13:58:49.848040 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846008    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 13:58:49.848556 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846011    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 13:58:49.848556 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846015    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 13:58:49.848556 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846024    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 13:58:49.848556 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846026    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 13:58:49.848556 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846029    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 13:58:49.848556 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846032    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 13:58:49.848556 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846034    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 13:58:49.848556 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846037    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 13:58:49.848556 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846040    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 13:58:49.848556 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846045    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 13:58:49.848556 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846049    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 13:58:49.848556 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846052    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 13:58:49.848556 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846055    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 13:58:49.848556 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846058    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 13:58:49.848556 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846061    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 13:58:49.848556 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846064    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 13:58:49.848556 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846067    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 13:58:49.848556 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846069    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 13:58:49.848556 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846072    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 13:58:49.849020 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846075    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 13:58:49.849020 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846077    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 13:58:49.849020 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846080    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 13:58:49.849020 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846083    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 13:58:49.849020 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846086    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 13:58:49.849020 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846088    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 13:58:49.849020 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846091    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 13:58:49.849020 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846093    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 13:58:49.849020 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846096    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 13:58:49.849020 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846098    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 13:58:49.849020 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846101    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 13:58:49.849020 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846103    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 13:58:49.849020 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846106    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 13:58:49.849020 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846108    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 13:58:49.849020 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846111    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 13:58:49.849020 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846113    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 13:58:49.849020 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846117    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 13:58:49.849020 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846121    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 13:58:49.849020 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846124    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 13:58:49.849512 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846127    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 13:58:49.849512 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846129    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 13:58:49.849512 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846132    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 13:58:49.849512 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846134    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 13:58:49.849512 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846138    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 13:58:49.849512 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846140    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 13:58:49.849512 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846143    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 13:58:49.849512 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846145    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 13:58:49.849512 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846148    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 13:58:49.849512 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846150    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 13:58:49.849512 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846153    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 13:58:49.849512 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846156    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 13:58:49.849512 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846158    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 13:58:49.849512 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846161    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 13:58:49.849512 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846163    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 13:58:49.849512 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846166    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 13:58:49.849512 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846184    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 13:58:49.849512 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846189    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 13:58:49.849512 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846192    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 13:58:49.849512 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846569    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 13:58:49.850007 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846573    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 13:58:49.850007 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846576    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 13:58:49.850007 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846580    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 13:58:49.850007 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846582    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 13:58:49.850007 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846585    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 13:58:49.850007 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846587    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 13:58:49.850007 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846590    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 13:58:49.850007 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846593    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 13:58:49.850007 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846595    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 13:58:49.850007 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846598    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 13:58:49.850007 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846601    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 13:58:49.850007 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846604    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 13:58:49.850007 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846607    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 13:58:49.850007 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846610    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 13:58:49.850007 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846613    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 13:58:49.850007 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846615    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 13:58:49.850007 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846619    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 13:58:49.850007 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846622    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 13:58:49.850007 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846624    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 13:58:49.850007 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846627    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 13:58:49.850518 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846630    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 13:58:49.850518 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846633    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 13:58:49.850518 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846635    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 13:58:49.850518 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846638    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 13:58:49.850518 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846640    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 13:58:49.850518 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846643    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 13:58:49.850518 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846645    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 13:58:49.850518 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846648    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 13:58:49.850518 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846650    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 13:58:49.850518 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846653    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 13:58:49.850518 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846656    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 13:58:49.850518 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846658    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 13:58:49.850518 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846662    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 13:58:49.850518 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846666    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 13:58:49.850518 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846668    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 13:58:49.850518 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846671    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 13:58:49.850518 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846673    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 13:58:49.850518 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846677    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 13:58:49.850518 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846680    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 13:58:49.851015 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846683    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 13:58:49.851015 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846686    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 13:58:49.851015 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846689    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 13:58:49.851015 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846693    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 13:58:49.851015 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846696    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 13:58:49.851015 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846699    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 13:58:49.851015 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846702    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 13:58:49.851015 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846705    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 13:58:49.851015 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846707    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 13:58:49.851015 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846710    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 13:58:49.851015 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846713    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 13:58:49.851015 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846716    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 13:58:49.851015 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846719    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 13:58:49.851015 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846721    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 13:58:49.851015 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846724    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 13:58:49.851015 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846727    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 13:58:49.851015 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846729    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 13:58:49.851015 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846732    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 13:58:49.851015 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846734    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 13:58:49.851015 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846737    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 13:58:49.851538 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846739    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 13:58:49.851538 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846742    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 13:58:49.851538 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846744    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 13:58:49.851538 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846747    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 13:58:49.851538 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846750    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 13:58:49.851538 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846752    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 13:58:49.851538 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846755    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 13:58:49.851538 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846757    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 13:58:49.851538 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846760    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 13:58:49.851538 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846762    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 13:58:49.851538 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846765    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 13:58:49.851538 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846768    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 13:58:49.851538 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846770    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 13:58:49.851538 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846774    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 13:58:49.851538 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846777    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 13:58:49.851538 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846779    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 13:58:49.851538 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846787    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 13:58:49.851538 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846790    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 13:58:49.851538 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846793    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 13:58:49.852015 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846796    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 13:58:49.852015 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846799    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 13:58:49.852015 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846803    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 13:58:49.852015 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846805    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 13:58:49.852015 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846808    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 13:58:49.852015 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846811    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 13:58:49.852015 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846813    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 13:58:49.852015 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846816    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 13:58:49.852015 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846818    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 13:58:49.852015 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846821    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 13:58:49.852015 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846824    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 13:58:49.852015 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846826    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 13:58:49.852015 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846829    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 13:58:49.852015 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846831    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 13:58:49.852015 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846834    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 13:58:49.852015 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846836    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 13:58:49.852015 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846839    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 13:58:49.852015 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846841    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 13:58:49.852015 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846845    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 13:58:49.852015 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.846847    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 13:58:49.852015 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.846919    2579 flags.go:64] FLAG: --address="0.0.0.0"
Sep 18 13:58:49.852550 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.846954    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 18 13:58:49.852550 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.846960    2579 flags.go:64] FLAG: --anonymous-auth="true"
Sep 18 13:58:49.852550 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.846964    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 18 13:58:49.852550 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.846968    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 18 13:58:49.852550 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.846971    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 18 13:58:49.852550 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.846980    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 18 13:58:49.852550 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.846984    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 18 13:58:49.852550 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.846988    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 18 13:58:49.852550 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.846992    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 18 13:58:49.852550 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.846997    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 18 13:58:49.852550 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847000    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 18 13:58:49.852550 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847003    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 18 13:58:49.852550 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847006    2579 flags.go:64] FLAG: --cgroup-root=""
Sep 18 13:58:49.852550 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847009    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 18 13:58:49.852550 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847012    2579 flags.go:64] FLAG: --client-ca-file=""
Sep 18 13:58:49.852550 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847015    2579 flags.go:64] FLAG: --cloud-provider="external"
Sep 18 13:58:49.852550 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847018    2579 flags.go:64] FLAG: --cluster-dns="[]"
Sep 18 13:58:49.852550 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847022    2579 flags.go:64] FLAG: --cluster-domain=""
Sep 18 13:58:49.852550 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847025    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 18 13:58:49.852550 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847028    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 18 13:58:49.852550 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847031    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 18 13:58:49.852550 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847037    2579 flags.go:64] FLAG: --container-log-max-files="5"
Sep 18 13:58:49.852550 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847041    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 18 13:58:49.852550 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847044    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 18 13:58:49.853126 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847047    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 18 13:58:49.853126 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847051    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 18 13:58:49.853126 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847053    2579 flags.go:64] FLAG: --contention-profiling="false"
Sep 18 13:58:49.853126 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847056    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 18 13:58:49.853126 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847059    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 18 13:58:49.853126 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847063    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 18 13:58:49.853126 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847065    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 18 13:58:49.853126 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847069    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 18 13:58:49.853126 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847072    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 18 13:58:49.853126 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847075    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 18 13:58:49.853126 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847078    2579 flags.go:64] FLAG: --enable-load-reader="false"
Sep 18 13:58:49.853126 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847081    2579 flags.go:64] FLAG: --enable-server="true"
Sep 18 13:58:49.853126 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847084    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 18 13:58:49.853126 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847092    2579 flags.go:64] FLAG: --event-burst="100"
Sep 18 13:58:49.853126 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847095    2579 flags.go:64] FLAG: --event-qps="50"
Sep 18 13:58:49.853126 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847098    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 18 13:58:49.853126 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847102    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 18 13:58:49.853126 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847105    2579 flags.go:64] FLAG: --eviction-hard=""
Sep 18 13:58:49.853126 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847109    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 18 13:58:49.853126 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847112    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 18 13:58:49.853126 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847115    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 18 13:58:49.853126 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847118    2579 flags.go:64] FLAG: --eviction-soft=""
Sep 18 13:58:49.853126 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847121    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 18 13:58:49.853126 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847124    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 18 13:58:49.853126 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847127    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 18 13:58:49.853797 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847130    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 18 13:58:49.853797 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847133    2579 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 18 13:58:49.853797 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847136    2579 flags.go:64] FLAG: --fail-swap-on="true"
Sep 18 13:58:49.853797 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847139    2579 flags.go:64] FLAG: --feature-gates=""
Sep 18 13:58:49.853797 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847144    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 18 13:58:49.853797 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847147    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 18 13:58:49.853797 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847150    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 18 13:58:49.853797 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847153    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 18 13:58:49.853797 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847156    2579 flags.go:64] FLAG: --healthz-port="10248"
Sep 18 13:58:49.853797 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847159    2579 flags.go:64] FLAG: --help="false"
Sep 18 13:58:49.853797 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847162    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-143-120.ec2.internal"
Sep 18 13:58:49.853797 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847166    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 18 13:58:49.853797 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847182    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 18 13:58:49.853797 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847185    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 18 13:58:49.853797 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847189    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 18 13:58:49.853797 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847192    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 18 13:58:49.853797 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847195    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 18 13:58:49.853797 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847198    2579 flags.go:64] FLAG: --image-service-endpoint=""
Sep 18 13:58:49.853797 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847201    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 18 13:58:49.853797 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847204    2579 flags.go:64] FLAG: --kube-api-burst="100"
Sep 18 13:58:49.853797 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847207    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 18 13:58:49.853797 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847210    2579 flags.go:64] FLAG: --kube-api-qps="50"
Sep 18 13:58:49.853797 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847212    2579 flags.go:64] FLAG: --kube-reserved=""
Sep 18 13:58:49.853797 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847216    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 18 13:58:49.854434 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847218    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 18 13:58:49.854434 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847222    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 18 13:58:49.854434 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847225    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 18 13:58:49.854434 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847228    2579 flags.go:64] FLAG: --lock-file=""
Sep 18 13:58:49.854434 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847231    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 18 13:58:49.854434 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847234    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 18 13:58:49.854434 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847238    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 18 13:58:49.854434 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847244    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 18 13:58:49.854434 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847246    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 18 13:58:49.854434 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847249    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 18 13:58:49.854434 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847252    2579 flags.go:64] FLAG: --logging-format="text"
Sep 18 13:58:49.854434 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847255    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 18 13:58:49.854434 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847260    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 18 13:58:49.854434 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847263    2579 flags.go:64] FLAG: --manifest-url=""
Sep 18 13:58:49.854434 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847266    2579 flags.go:64] FLAG: --manifest-url-header=""
Sep 18 13:58:49.854434 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847270    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 18 13:58:49.854434 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847273    2579 flags.go:64] FLAG: --max-open-files="1000000"
Sep 18 13:58:49.854434 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847277    2579 flags.go:64] FLAG: --max-pods="110"
Sep 18 13:58:49.854434 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847280    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 18 13:58:49.854434 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847283    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 18 13:58:49.854434 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847286    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 18 13:58:49.854434 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847289    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 18 13:58:49.854434 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847292    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 18 13:58:49.854434 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847295    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 18 13:58:49.854434 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847298    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 18 13:58:49.855243 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847305    2579 flags.go:64] FLAG: --node-status-max-images="50"
Sep 18 13:58:49.855243 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847308    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 18 13:58:49.855243 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847316    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 18 13:58:49.855243 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847319    2579 flags.go:64] FLAG: --pod-cidr=""
Sep 18 13:58:49.855243 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847322    2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 18 13:58:49.855243 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847327    2579 flags.go:64] FLAG: --pod-manifest-path=""
Sep 18 13:58:49.855243 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847330    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 18 13:58:49.855243 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847333    2579 flags.go:64] FLAG: --pods-per-core="0"
Sep 18 13:58:49.855243 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847337    2579 flags.go:64] FLAG: --port="10250"
Sep 18 13:58:49.855243 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847340    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 18 13:58:49.855243 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847344    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0cba638713ff57ecd"
Sep 18 13:58:49.855243 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847347    2579 flags.go:64] FLAG: --qos-reserved=""
Sep 18 13:58:49.855243 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847350    2579 flags.go:64] FLAG: --read-only-port="10255"
Sep 18 13:58:49.855243 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847353    2579 flags.go:64] FLAG: --register-node="true"
Sep 18 13:58:49.855243 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847356    2579 flags.go:64] FLAG: --register-with-taints=""
Sep 18 13:58:49.855243 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847360    2579 flags.go:64] FLAG: --registry-burst="10"
Sep 18 13:58:49.855243 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847363    2579 flags.go:64] FLAG: --registry-qps="5"
Sep 18 13:58:49.855243 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847366    2579 flags.go:64] FLAG: --reserved-cpus=""
Sep 18 13:58:49.855243 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847369    2579 flags.go:64] FLAG: --reserved-memory=""
Sep 18 13:58:49.855243 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847374    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 18 13:58:49.855243 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847377    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 18 13:58:49.855243 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847380    2579 flags.go:64] FLAG: --rotate-certificates="false"
Sep 18 13:58:49.855243 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847383    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 18 13:58:49.855243 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847386    2579 flags.go:64] FLAG: --runonce="false"
Sep 18 13:58:49.855243 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847388    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 18 13:58:49.856030 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847392    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 18 13:58:49.856030 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847395    2579 flags.go:64] FLAG: --seccomp-default="false"
Sep 18 13:58:49.856030 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847398    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 18 13:58:49.856030 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847401    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 18 13:58:49.856030 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847404    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 18 13:58:49.856030 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847407    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 18 13:58:49.856030 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847410    2579 flags.go:64] FLAG: --storage-driver-password="root"
Sep 18 13:58:49.856030 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847413    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 18 13:58:49.856030 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847416    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 18 13:58:49.856030 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847419    2579 flags.go:64] FLAG: --storage-driver-user="root"
Sep 18 13:58:49.856030 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847421    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 18 13:58:49.856030 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847424    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 18 13:58:49.856030 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847427    2579 flags.go:64] FLAG: --system-cgroups=""
Sep 18 13:58:49.856030 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847430    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 18 13:58:49.856030 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847435    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 18 13:58:49.856030 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847438    2579 flags.go:64] FLAG: --tls-cert-file=""
Sep 18 13:58:49.856030 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847441    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 18 13:58:49.856030 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847445    2579 flags.go:64] FLAG: --tls-min-version=""
Sep 18 13:58:49.856030 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847448    2579 flags.go:64] FLAG: --tls-private-key-file=""
Sep 18 13:58:49.856030 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847451    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 18 13:58:49.856030 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847453    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 18 13:58:49.856030 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847456    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 18 13:58:49.856030 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847459    2579 flags.go:64] FLAG: --v="2"
Sep 18 13:58:49.856030 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847463    2579 flags.go:64] FLAG: --version="false"
Sep 18 13:58:49.856030 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847467    2579 flags.go:64] FLAG: --vmodule=""
Sep 18 13:58:49.856714 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847470    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 18 13:58:49.856714 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.847475    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 18 13:58:49.856714 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847587    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 13:58:49.856714 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847593    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 13:58:49.856714 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847596    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 13:58:49.856714 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847608    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 13:58:49.856714 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847612    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 13:58:49.856714 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847615    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 13:58:49.856714 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847617    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 13:58:49.856714 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847620    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 13:58:49.856714 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847623    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 13:58:49.856714 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847626    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 13:58:49.856714 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847629    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 13:58:49.856714 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847631    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 13:58:49.856714 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847634    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 13:58:49.856714 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847636    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 13:58:49.856714 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847639    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 13:58:49.856714 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847642    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 13:58:49.856714 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847645    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 13:58:49.856714 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847647    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 13:58:49.857222 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847650    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 13:58:49.857222 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847654    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 13:58:49.857222 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847657    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 13:58:49.857222 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847660    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 13:58:49.857222 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847662    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 13:58:49.857222 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847665    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 13:58:49.857222 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847669    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 13:58:49.857222 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847671    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 13:58:49.857222 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847675    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 13:58:49.857222 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847677    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 13:58:49.857222 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847680    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 13:58:49.857222 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847683    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 13:58:49.857222 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847685    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 13:58:49.857222 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847689    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 13:58:49.857222 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847692    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 13:58:49.857222 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847695    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 13:58:49.857222 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847698    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 13:58:49.857222 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847700    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 13:58:49.857222 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847703    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 13:58:49.857222 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847705    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 13:58:49.857716 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847708    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 13:58:49.857716 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847711    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 13:58:49.857716 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847714    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 13:58:49.857716 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847726    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 13:58:49.857716 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847729    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 13:58:49.857716 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847732    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 13:58:49.857716 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847735    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 13:58:49.857716 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847738    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 13:58:49.857716 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847741    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 13:58:49.857716 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847744    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 13:58:49.857716 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847746    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 13:58:49.857716 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847751    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 13:58:49.857716 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847754    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 13:58:49.857716 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847758    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 13:58:49.857716 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847761    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 13:58:49.857716 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847763    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 13:58:49.857716 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847766    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 13:58:49.857716 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847769    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 13:58:49.857716 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847772    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 13:58:49.858255 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847775    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 13:58:49.858255 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847778    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 13:58:49.858255 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847781    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 13:58:49.858255 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847784    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 13:58:49.858255 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847786    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 13:58:49.858255 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847789    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 13:58:49.858255 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847793    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 13:58:49.858255 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847796    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 13:58:49.858255 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847799    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 13:58:49.858255 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847802    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 13:58:49.858255 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847804    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 13:58:49.858255 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847807    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 13:58:49.858255 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847809    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 13:58:49.858255 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847812    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 13:58:49.858255 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847815    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 13:58:49.858255 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847817    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 13:58:49.858255 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847820    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 13:58:49.858255 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847823    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 13:58:49.858255 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847825    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 13:58:49.858255 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847828    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 13:58:49.858750 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847830    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 13:58:49.858750 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847833    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 13:58:49.858750 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847836    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 13:58:49.858750 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847839    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 13:58:49.858750 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847841    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 13:58:49.858750 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847845    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 13:58:49.858750 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847848    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 13:58:49.858750 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847851    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 13:58:49.858750 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847854    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 13:58:49.858750 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847857    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 13:58:49.858750 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847859    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 13:58:49.858750 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847862    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 13:58:49.858750 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847864    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 13:58:49.858750 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847868    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 13:58:49.858750 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847871    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 13:58:49.858750 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847873    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 13:58:49.858750 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847876    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 13:58:49.858750 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847878    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 13:58:49.858750 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847882    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 13:58:49.859227 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847885    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 13:58:49.859227 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847887    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 13:58:49.859227 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.847890    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 13:58:49.859227 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.848367    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 13:58:49.859227 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.857274    2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 18 13:58:49.859227 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.857290    2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 18 13:58:49.859227 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.857310    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 18 13:58:49.859227 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.857316    2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 18 13:58:49.859227 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857377    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 13:58:49.859227 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857383    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 13:58:49.859227 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857386    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 13:58:49.859227 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857389    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 13:58:49.859227 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857392    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 13:58:49.859227 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857395    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 13:58:49.859227 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857398    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 13:58:49.859619 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857401    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 13:58:49.859619 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857404    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 13:58:49.859619 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857406    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 13:58:49.859619 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857409    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 13:58:49.859619 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857413    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 13:58:49.859619 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857415    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 13:58:49.859619 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857418    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 13:58:49.859619 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857421    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 13:58:49.859619 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857424    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 13:58:49.859619 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857430    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 13:58:49.859619 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857433    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 13:58:49.859619 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857436    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 13:58:49.859619 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857439    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 13:58:49.859619 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857441    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 13:58:49.859619 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857444    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 13:58:49.859619 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857447    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 13:58:49.859619 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857450    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 13:58:49.859619 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857452    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 13:58:49.859619 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857455    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 13:58:49.859619 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857458    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 13:58:49.860115 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857460    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 13:58:49.860115 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857463    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 13:58:49.860115 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857466    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 13:58:49.860115 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857469    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 13:58:49.860115 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857473    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 13:58:49.860115 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857476    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 13:58:49.860115 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857478    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 13:58:49.860115 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857481    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 13:58:49.860115 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857484    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 13:58:49.860115 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857487    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 13:58:49.860115 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857490    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 13:58:49.860115 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857493    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 13:58:49.860115 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857495    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 13:58:49.860115 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857498    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 13:58:49.860115 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857501    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 13:58:49.860115 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857503    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 13:58:49.860115 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857506    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 13:58:49.860115 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857509    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 13:58:49.860115 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857511    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 13:58:49.860115 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857515    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 13:58:49.860666 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857518    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 13:58:49.860666 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857521    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 13:58:49.860666 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857524    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 13:58:49.860666 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857527    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 13:58:49.860666 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857529    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 13:58:49.860666 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857532    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 13:58:49.860666 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857535    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 13:58:49.860666 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857539    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 13:58:49.860666 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857543    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 13:58:49.860666 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857546    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 13:58:49.860666 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857549    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 13:58:49.860666 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857551    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 13:58:49.860666 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857554    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 13:58:49.860666 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857557    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 13:58:49.860666 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857560    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 13:58:49.860666 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857563    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 13:58:49.860666 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857566    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 13:58:49.860666 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857569    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 13:58:49.860666 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857572    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 13:58:49.861144 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857574    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 13:58:49.861144 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857577    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 13:58:49.861144 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857580    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 13:58:49.861144 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857583    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 13:58:49.861144 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857586    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 13:58:49.861144 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857588    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 13:58:49.861144 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857591    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 13:58:49.861144 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857594    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 13:58:49.861144 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857597    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 13:58:49.861144 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857599    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 13:58:49.861144 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857602    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 13:58:49.861144 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857604    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 13:58:49.861144 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857608    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 13:58:49.861144 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857614    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 13:58:49.861144 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857618    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 13:58:49.861144 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857622    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 13:58:49.861144 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857625    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 13:58:49.861144 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857629    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 13:58:49.861144 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857632    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 13:58:49.861884 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857635    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 13:58:49.861884 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857638    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 13:58:49.861884 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857641    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 13:58:49.861884 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857643    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 13:58:49.861884 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857646    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 13:58:49.861884 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857648    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 13:58:49.861884 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857651    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 13:58:49.861884 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857654    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 13:58:49.861884 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857657    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 13:58:49.861884 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857660    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 13:58:49.861884 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857663    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 13:58:49.861884 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857665    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 13:58:49.861884 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857668    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 13:58:49.861884 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857670    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 13:58:49.861884 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.857676    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 13:58:49.862594 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857791    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 13:58:49.862594 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857797    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 13:58:49.862594 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857800    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 13:58:49.862594 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857803    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 13:58:49.862594 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857806    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 13:58:49.862594 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857809    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 13:58:49.862594 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857812    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 13:58:49.862594 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857814    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 13:58:49.862594 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857817    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 13:58:49.862594 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857820    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 13:58:49.862594 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857823    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 13:58:49.862594 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857826    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 13:58:49.862594 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857830    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 13:58:49.862594 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857833    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 13:58:49.862594 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857836    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 13:58:49.862594 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857839    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 13:58:49.862594 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857842    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 13:58:49.862594 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857846    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 13:58:49.862594 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857848    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 13:58:49.862594 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857851    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 13:58:49.863086 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857855    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 13:58:49.863086 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857859    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 13:58:49.863086 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857862    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 13:58:49.863086 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857865    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 13:58:49.863086 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857867    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 13:58:49.863086 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857870    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 13:58:49.863086 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857873    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 13:58:49.863086 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857876    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 13:58:49.863086 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857878    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 13:58:49.863086 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857881    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 13:58:49.863086 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857883    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 13:58:49.863086 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857886    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 13:58:49.863086 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857888    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 13:58:49.863086 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857891    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 13:58:49.863086 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857894    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 13:58:49.863086 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857896    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 13:58:49.863086 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857899    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 13:58:49.863086 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857901    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 13:58:49.863086 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857904    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 13:58:49.863703 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857907    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 13:58:49.863703 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857909    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 13:58:49.863703 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857912    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 13:58:49.863703 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857914    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 13:58:49.863703 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857917    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 13:58:49.863703 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857920    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 13:58:49.863703 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857923    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 13:58:49.863703 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857926    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 13:58:49.863703 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857928    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 13:58:49.863703 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857932    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 13:58:49.863703 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857934    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 13:58:49.863703 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857937    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 13:58:49.863703 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857939    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 13:58:49.863703 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857942    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 13:58:49.863703 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857945    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 13:58:49.863703 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857947    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 13:58:49.863703 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857950    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 13:58:49.863703 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857952    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 13:58:49.863703 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857955    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 13:58:49.863703 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857958    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 13:58:49.864259 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857960    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 13:58:49.864259 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857963    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 13:58:49.864259 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857966    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 13:58:49.864259 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857969    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 13:58:49.864259 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857971    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 13:58:49.864259 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857974    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 13:58:49.864259 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857976    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 13:58:49.864259 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857979    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 13:58:49.864259 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857982    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 13:58:49.864259 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857984    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 13:58:49.864259 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857988    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 13:58:49.864259 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857991    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 13:58:49.864259 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857994    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 13:58:49.864259 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.857997    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 13:58:49.864259 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.858000    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 13:58:49.864259 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.858002    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 13:58:49.864259 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.858005    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 13:58:49.864259 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.858007    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 13:58:49.864259 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.858010    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 13:58:49.864259 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.858013    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 13:58:49.864891 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.858016    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 13:58:49.864891 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.858018    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 13:58:49.864891 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.858022    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 13:58:49.864891 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.858025    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 13:58:49.864891 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.858029    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 13:58:49.864891 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.858032    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 13:58:49.864891 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.858035    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 13:58:49.864891 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.858038    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 13:58:49.864891 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.858040    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 13:58:49.864891 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.858043    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 13:58:49.864891 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.858046    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 13:58:49.864891 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.858049    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 13:58:49.864891 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.858052    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 13:58:49.864891 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.858054    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 13:58:49.864891 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.858057    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 13:58:49.864891 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.858060    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 13:58:49.864891 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.858062    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 13:58:49.864891 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.858065    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 13:58:49.864891 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.858068    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 13:58:49.865542 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:49.858070    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 13:58:49.865542 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.858076    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 13:58:49.865542 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.858209    2579 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 18 13:58:49.865542 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.860198    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 18 13:58:49.865542 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.861200    2579 server.go:1019] "Starting client certificate rotation"
Sep 18 13:58:49.865542 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.861309    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 13:58:49.865542 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.861362    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 13:58:49.865542 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.863552    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 13:58:49.865542 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.864774    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 13:58:49.868566 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.868550    2579 log.go:25] "Validated CRI v1 runtime API"
Sep 18 13:58:49.873102 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.873087    2579 log.go:25] "Validated CRI v1 image API"
Sep 18 13:58:49.874389 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.874374    2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 18 13:58:49.877329 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.877310    2579 fs.go:135] Filesystem UUIDs: map[29e0a75d-9700-4717-8f93-73a49c045b1c:/dev/nvme0n1p3 759fa88b-7d2c-441d-a750-788dfff1f527:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 18 13:58:49.877392 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.877329    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 18 13:58:49.882286 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.882151    2579 manager.go:216] Machine: {Timestamp:2026-09-18 13:58:49.880820414 +0000 UTC m=+1.769536336 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097256 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec247752d266270ff08bff64898de5fa SystemUUID:ec247752-d266-270f-f08b-ff64898de5fa BootID:c4111530-1092-4e53-9ca2-50d40b7215fd Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d1:45:41:b9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d1:45:41:b9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ce:05:e1:d8:d5:32 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 18 13:58:49.882286 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.882279    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 18 13:58:49.882395 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.882384    2579 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 18 13:58:49.884220 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.884140    2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 18 13:58:49.884530 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.884390    2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-120.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 18 13:58:49.885570 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.885558    2579 topology_manager.go:138] "Creating topology manager with none policy"
Sep 18 13:58:49.885609 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.885573    2579 container_manager_linux.go:309] "Creating device plugin manager"
Sep 18 13:58:49.885609 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.885581    2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 13:58:49.885609 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.885598    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 13:58:49.885691 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.885666    2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 18 13:58:49.889184 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.889159    2579 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 13:58:49.889317 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.889307    2579 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 18 13:58:49.889349 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.889346    2579 kubelet.go:486] "Attempting to sync node with API server"
Sep 18 13:58:49.889384 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.889355    2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 18 13:58:49.889384 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.889365    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 18 13:58:49.889384 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.889373    2579 kubelet.go:398] "Adding apiserver pod source"
Sep 18 13:58:49.889384 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.889381    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 18 13:58:49.889667 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.889656    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 13:58:49.889713 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.889673    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 13:58:49.891875 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.891852    2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 18 13:58:49.893887 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.893873    2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 18 13:58:49.893940 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.893903    2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 18 13:58:49.894943 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.894933    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 18 13:58:49.894984 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.894948    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 18 13:58:49.894984 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.894954    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 18 13:58:49.894984 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.894960    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 18 13:58:49.894984 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.894968    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 18 13:58:49.894984 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.894974    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 18 13:58:49.895108 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.894987    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 18 13:58:49.895108 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.894993    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 18 13:58:49.895108 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.895000    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 18 13:58:49.895108 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.895005    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 18 13:58:49.895108 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.895014    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 18 13:58:49.895108 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.895019    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 18 13:58:49.895108 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.895041    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 18 13:58:49.895108 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.895046    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 18 13:58:49.898460 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.898440    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 13:58:49.898605 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.898593    2579 server.go:1268] "Started kubelet"
Sep 18 13:58:49.898703 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.898675    2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 18 13:58:49.898752 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.898693    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 18 13:58:49.898797 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.898758    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 18 13:58:49.899387 ip-10-0-143-120 systemd[1]: Started Kubernetes Kubelet.
Sep 18 13:58:49.899829 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.899813    2579 server.go:310] "Adding debug handlers to kubelet server"
Sep 18 13:58:49.899918 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.899875    2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 18 13:58:49.903075 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:49.903055    2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 18 13:58:49.904883 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.904868    2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 18 13:58:49.904883 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.904880    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 18 13:58:49.905017 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:49.904937    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-120.ec2.internal\" not found"
Sep 18 13:58:49.905457 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.905439    2579 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 18 13:58:49.905648 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.905622    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 18 13:58:49.905736 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.905717    2579 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 18 13:58:49.905953 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.905936    2579 factory.go:153] Registering CRI-O factory
Sep 18 13:58:49.905953 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.905953    2579 factory.go:223] Registration of the crio container factory successfully
Sep 18 13:58:49.906065 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.906004    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 18 13:58:49.906065 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.906012    2579 factory.go:55] Registering systemd factory
Sep 18 13:58:49.906065 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.906021    2579 factory.go:223] Registration of the systemd container factory successfully
Sep 18 13:58:49.906065 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.906036    2579 factory.go:103] Registering Raw factory
Sep 18 13:58:49.906541 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.906526    2579 reconstruct.go:97] "Volume reconstruction finished"
Sep 18 13:58:49.906640 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.906622    2579 reconciler.go:29] "Reconciler: start to sync state"
Sep 18 13:58:49.906640 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.906614    2579 manager.go:1196] Started watching for new ooms in manager
Sep 18 13:58:49.906876 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:49.906631    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-120.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 13:58:49.906970 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:49.906948    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 13:58:49.907013 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.906991    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-120.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 13:58:49.907083 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.907072    2579 manager.go:349] Starting recovery of all containers
Sep 18 13:58:49.908744 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:49.908718    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 13:58:49.909657 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:49.909494    2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-120.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 18 13:58:49.938428 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:49.906564    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-120.ec2.internal.18d66efbd9ef5a12  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-120.ec2.internal,UID:ip-10-0-143-120.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-120.ec2.internal,},FirstTimestamp:2026-09-18 13:58:49.898539538 +0000 UTC m=+1.787255463,LastTimestamp:2026-09-18 13:58:49.898539538 +0000 UTC m=+1.787255463,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-120.ec2.internal,}"
Sep 18 13:58:49.938977 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.938931    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-x82fh"
Sep 18 13:58:49.939052 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.938982    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-x82fh"
Sep 18 13:58:49.946357 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.946341    2579 manager.go:354] Recovery completed
Sep 18 13:58:49.949955 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.949942    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 13:58:49.951580 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.951565    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-120.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 13:58:49.951726 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.951594    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-120.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 13:58:49.951726 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.951617    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-120.ec2.internal" event="NodeHasSufficientPID"
Sep 18 13:58:49.952121 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.952104    2579 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 18 13:58:49.952121 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.952120    2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 18 13:58:49.952246 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.952136    2579 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 13:58:49.952246 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.952107    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 13:58:49.952822 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.952809    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-120.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 13:58:49.952865 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.952836    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-120.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 13:58:49.952865 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.952851    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-120.ec2.internal" event="NodeHasSufficientPID"
Sep 18 13:58:49.953521 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:49.953463    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-120.ec2.internal.18d66efbdd18aab1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-120.ec2.internal,UID:ip-10-0-143-120.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-120.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-120.ec2.internal,},FirstTimestamp:2026-09-18 13:58:49.951578801 +0000 UTC m=+1.840294727,LastTimestamp:2026-09-18 13:58:49.951578801 +0000 UTC m=+1.840294727,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-120.ec2.internal,}"
Sep 18 13:58:49.954096 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.954085    2579 policy_none.go:49] "None policy: Start"
Sep 18 13:58:49.954153 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.954102    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 18 13:58:49.954153 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.954113    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 18 13:58:49.955963 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:49.955950    2579 policy_none.go:47] "Start"
Sep 18 13:58:50.002727 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.002710    2579 manager.go:337] "Starting Device Plugin manager"
Sep 18 13:58:50.006182 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:50.002740    2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 18 13:58:50.006182 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.002748    2579 server.go:87] "Starting device plugin registration server"
Sep 18 13:58:50.006182 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.002955    2579 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 18 13:58:50.006182 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.002977    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 18 13:58:50.006182 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.003031    2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 18 13:58:50.006182 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.003094    2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 18 13:58:50.006182 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.003100    2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 18 13:58:50.006182 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:50.003489    2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 18 13:58:50.006182 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:50.003540    2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-120.ec2.internal\" not found"
Sep 18 13:58:50.027614 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.027582    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 18 13:58:50.028756 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.028734    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 18 13:58:50.028839 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.028776    2579 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 18 13:58:50.028839 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.028796    2579 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 18 13:58:50.028936 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:50.028829    2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 18 13:58:50.031915 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.031899    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 13:58:50.103656 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.103596    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 13:58:50.104384 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.104368    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-120.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 13:58:50.104451 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.104397    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-120.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 13:58:50.104451 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.104413    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-120.ec2.internal" event="NodeHasSufficientPID"
Sep 18 13:58:50.104451 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.104437    2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-120.ec2.internal"
Sep 18 13:58:50.110510 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.110496    2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-120.ec2.internal"
Sep 18 13:58:50.110578 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:50.110513    2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-120.ec2.internal\": node \"ip-10-0-143-120.ec2.internal\" not found"
Sep 18 13:58:50.128733 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:50.128712    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-120.ec2.internal\" not found"
Sep 18 13:58:50.129753 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.129736    2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-120.ec2.internal"]
Sep 18 13:58:50.129823 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.129802    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 13:58:50.130527 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.130512    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-120.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 13:58:50.130588 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.130539    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-120.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 13:58:50.130588 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.130549    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-120.ec2.internal" event="NodeHasSufficientPID"
Sep 18 13:58:50.132624 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.132612    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 13:58:50.132745 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.132731    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal"
Sep 18 13:58:50.132811 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.132766    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 13:58:50.133290 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.133277    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-120.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 13:58:50.133363 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.133289    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-120.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 13:58:50.133363 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.133299    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-120.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 13:58:50.133363 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.133306    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-120.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 13:58:50.133363 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.133316    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-120.ec2.internal" event="NodeHasSufficientPID"
Sep 18 13:58:50.133483 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.133316    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-120.ec2.internal" event="NodeHasSufficientPID"
Sep 18 13:58:50.135405 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.135389    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-120.ec2.internal"
Sep 18 13:58:50.135489 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.135430    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 13:58:50.136046 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.136027    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-120.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 13:58:50.136118 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.136060    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-120.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 13:58:50.136118 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.136074    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-120.ec2.internal" event="NodeHasSufficientPID"
Sep 18 13:58:50.157793 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:50.157776    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-120.ec2.internal\" not found" node="ip-10-0-143-120.ec2.internal"
Sep 18 13:58:50.161968 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:50.161950    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-120.ec2.internal\" not found" node="ip-10-0-143-120.ec2.internal"
Sep 18 13:58:50.208117 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.208096    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/596255870e755bf312825ba49bcccfdb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal\" (UID: \"596255870e755bf312825ba49bcccfdb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal"
Sep 18 13:58:50.208208 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.208119    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/596255870e755bf312825ba49bcccfdb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal\" (UID: \"596255870e755bf312825ba49bcccfdb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal"
Sep 18 13:58:50.208208 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.208146    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/59194d24519fb418bb58aa767b6c58b0-config\") pod \"kube-apiserver-proxy-ip-10-0-143-120.ec2.internal\" (UID: \"59194d24519fb418bb58aa767b6c58b0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-120.ec2.internal"
Sep 18 13:58:50.229449 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:50.229423    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-120.ec2.internal\" not found"
Sep 18 13:58:50.308762 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.308729    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/596255870e755bf312825ba49bcccfdb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal\" (UID: \"596255870e755bf312825ba49bcccfdb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal"
Sep 18 13:58:50.308762 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.308759    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/596255870e755bf312825ba49bcccfdb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal\" (UID: \"596255870e755bf312825ba49bcccfdb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal"
Sep 18 13:58:50.308953 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.308777    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/59194d24519fb418bb58aa767b6c58b0-config\") pod \"kube-apiserver-proxy-ip-10-0-143-120.ec2.internal\" (UID: \"59194d24519fb418bb58aa767b6c58b0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-120.ec2.internal"
Sep 18 13:58:50.308953 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.308822    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/596255870e755bf312825ba49bcccfdb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal\" (UID: \"596255870e755bf312825ba49bcccfdb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal"
Sep 18 13:58:50.308953 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.308836    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/596255870e755bf312825ba49bcccfdb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal\" (UID: \"596255870e755bf312825ba49bcccfdb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal"
Sep 18 13:58:50.308953 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.308806    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/59194d24519fb418bb58aa767b6c58b0-config\") pod \"kube-apiserver-proxy-ip-10-0-143-120.ec2.internal\" (UID: \"59194d24519fb418bb58aa767b6c58b0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-120.ec2.internal"
Sep 18 13:58:50.329887 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:50.329855    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-120.ec2.internal\" not found"
Sep 18 13:58:50.430803 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:50.430744    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-120.ec2.internal\" not found"
Sep 18 13:58:50.460964 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.460927    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal"
Sep 18 13:58:50.464847 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.464820    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-120.ec2.internal"
Sep 18 13:58:50.531259 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:50.531215    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-120.ec2.internal\" not found"
Sep 18 13:58:50.631844 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:50.631806    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-120.ec2.internal\" not found"
Sep 18 13:58:50.732359 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:50.732274    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-120.ec2.internal\" not found"
Sep 18 13:58:50.832961 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:50.832927    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-120.ec2.internal\" not found"
Sep 18 13:58:50.861089 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.861067    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 18 13:58:50.861467 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.861225    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 13:58:50.918591 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.918570    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 13:58:50.944538 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.944493    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 13:53:49 +0000 UTC" deadline="2028-05-06 00:04:52.799129157 +0000 UTC"
Sep 18 13:58:50.944538 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:50.944537    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14290h6m1.854596475s"
Sep 18 13:58:51.005945 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.005825    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal"
Sep 18 13:58:51.007587 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:51.007560    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod596255870e755bf312825ba49bcccfdb.slice/crio-1ccbb8892b92d84d8d8e9af8d15109781680ce1f494d468c3e12f5c300139603 WatchSource:0}: Error finding container 1ccbb8892b92d84d8d8e9af8d15109781680ce1f494d468c3e12f5c300139603: Status 404 returned error can't find the container with id 1ccbb8892b92d84d8d8e9af8d15109781680ce1f494d468c3e12f5c300139603
Sep 18 13:58:51.007989 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:51.007971    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59194d24519fb418bb58aa767b6c58b0.slice/crio-de1faf8dc5dc88d7640253af77a2770f5c843309161c668ce2a3e0df8819937a WatchSource:0}: Error finding container de1faf8dc5dc88d7640253af77a2770f5c843309161c668ce2a3e0df8819937a: Status 404 returned error can't find the container with id de1faf8dc5dc88d7640253af77a2770f5c843309161c668ce2a3e0df8819937a
Sep 18 13:58:51.012968 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.012943    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 13:58:51.017050 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.017035    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 13:58:51.018855 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.018839    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-120.ec2.internal"
Sep 18 13:58:51.027503 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.027485    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 13:58:51.031645 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.031599    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal" event={"ID":"596255870e755bf312825ba49bcccfdb","Type":"ContainerStarted","Data":"1ccbb8892b92d84d8d8e9af8d15109781680ce1f494d468c3e12f5c300139603"}
Sep 18 13:58:51.032636 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.032616    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-120.ec2.internal" event={"ID":"59194d24519fb418bb58aa767b6c58b0","Type":"ContainerStarted","Data":"de1faf8dc5dc88d7640253af77a2770f5c843309161c668ce2a3e0df8819937a"}
Sep 18 13:58:51.259382 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.259317    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 13:58:51.424485 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.424461    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 13:58:51.890796 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.890773    2579 apiserver.go:52] "Watching apiserver"
Sep 18 13:58:51.896587 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.896563    2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 18 13:58:51.897043 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.897015    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-sg4tv","kube-system/global-pull-secret-syncer-2hvmm","openshift-dns/node-resolver-t9ks5","openshift-image-registry/node-ca-8n79p","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal","openshift-multus/multus-czbmm","openshift-multus/network-metrics-daemon-2mgfl","openshift-network-diagnostics/network-check-target-j7msq","openshift-network-operator/iptables-alerter-469c7","kube-system/konnectivity-agent-fkrgg","kube-system/kube-apiserver-proxy-ip-10-0-143-120.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnrrz","openshift-cluster-node-tuning-operator/tuned-bthf7","openshift-multus/multus-additional-cni-plugins-p5f5h"]
Sep 18 13:58:51.899957 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.899693    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2mgfl"
Sep 18 13:58:51.899957 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:51.899780    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2mgfl" podUID="57282ff1-fab4-4cd4-8087-5517bd249f10"
Sep 18 13:58:51.902088 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.902063    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7msq"
Sep 18 13:58:51.902205 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:51.902136    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j7msq" podUID="7e8de495-7804-4c38-af57-655459217f1c"
Sep 18 13:58:51.904210 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.904190    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t9ks5"
Sep 18 13:58:51.906616 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.906280    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 18 13:58:51.906616 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.906463    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8n79p"
Sep 18 13:58:51.908263 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.908240    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-g7mbb\""
Sep 18 13:58:51.908263 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.908254    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 18 13:58:51.908411 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.908247    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 18 13:58:51.909050 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.908771    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:51.909621 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.909482    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 18 13:58:51.909621 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.909485    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 18 13:58:51.910143 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.910124    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zlvr9\""
Sep 18 13:58:51.910257 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.910186    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 18 13:58:51.911111 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.911095    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-czbmm"
Sep 18 13:58:51.911578 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.911561    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-27qgb\""
Sep 18 13:58:51.911688 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.911668    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 18 13:58:51.911745 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.911672    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 18 13:58:51.911996 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.911870    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 18 13:58:51.912620 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.912392    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 18 13:58:51.912620 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.912443    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 13:58:51.912620 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.912581    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 18 13:58:51.913764 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.913739    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 18 13:58:51.913946 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.913910    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-469c7"
Sep 18 13:58:51.915720 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.915699    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/690b430b-1984-4119-beae-85786c10b267-hosts-file\") pod \"node-resolver-t9ks5\" (UID: \"690b430b-1984-4119-beae-85786c10b267\") " pod="openshift-dns/node-resolver-t9ks5"
Sep 18 13:58:51.915811 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.915733    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfq4k\" (UniqueName: \"kubernetes.io/projected/dbb463f3-8dcd-413d-859f-5dcf87c59115-kube-api-access-mfq4k\") pod \"node-ca-8n79p\" (UID: \"dbb463f3-8dcd-413d-859f-5dcf87c59115\") " pod="openshift-image-registry/node-ca-8n79p"
Sep 18 13:58:51.915811 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.915760    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4626a667-4cd6-468c-8ef3-6e3a925c2608-host-slash\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:51.915811 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.915792    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4626a667-4cd6-468c-8ef3-6e3a925c2608-etc-openvswitch\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:51.915980 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.915816    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r7hs\" (UniqueName: \"kubernetes.io/projected/4626a667-4cd6-468c-8ef3-6e3a925c2608-kube-api-access-7r7hs\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:51.915980 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.915839    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/690b430b-1984-4119-beae-85786c10b267-tmp-dir\") pod \"node-resolver-t9ks5\" (UID: \"690b430b-1984-4119-beae-85786c10b267\") " pod="openshift-dns/node-resolver-t9ks5"
Sep 18 13:58:51.915980 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.915880    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4626a667-4cd6-468c-8ef3-6e3a925c2608-host-kubelet\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:51.915980 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.915922    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4626a667-4cd6-468c-8ef3-6e3a925c2608-run-openvswitch\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:51.915980 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.915955    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4626a667-4cd6-468c-8ef3-6e3a925c2608-host-run-ovn-kubernetes\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:51.916256 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.916075    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4626a667-4cd6-468c-8ef3-6e3a925c2608-run-ovn\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:51.916256 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.916127    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4626a667-4cd6-468c-8ef3-6e3a925c2608-host-cni-bin\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:51.916256 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.916151    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4626a667-4cd6-468c-8ef3-6e3a925c2608-ovn-node-metrics-cert\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:51.916256 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.916209    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4626a667-4cd6-468c-8ef3-6e3a925c2608-ovnkube-script-lib\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:51.916482 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.916246    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbd24\" (UniqueName: \"kubernetes.io/projected/57282ff1-fab4-4cd4-8087-5517bd249f10-kube-api-access-dbd24\") pod \"network-metrics-daemon-2mgfl\" (UID: \"57282ff1-fab4-4cd4-8087-5517bd249f10\") " pod="openshift-multus/network-metrics-daemon-2mgfl"
Sep 18 13:58:51.916482 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.916287    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4626a667-4cd6-468c-8ef3-6e3a925c2608-var-lib-openvswitch\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:51.916482 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.916311    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4626a667-4cd6-468c-8ef3-6e3a925c2608-node-log\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:51.916482 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.916363    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4626a667-4cd6-468c-8ef3-6e3a925c2608-log-socket\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:51.916482 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.916388    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t8sk\" (UniqueName: \"kubernetes.io/projected/7e8de495-7804-4c38-af57-655459217f1c-kube-api-access-4t8sk\") pod \"network-check-target-j7msq\" (UID: \"7e8de495-7804-4c38-af57-655459217f1c\") " pod="openshift-network-diagnostics/network-check-target-j7msq"
Sep 18 13:58:51.916482 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.916420    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 18 13:58:51.916482 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.916458    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-nb7gc\""
Sep 18 13:58:51.916854 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.916413    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zndv4\" (UniqueName: \"kubernetes.io/projected/690b430b-1984-4119-beae-85786c10b267-kube-api-access-zndv4\") pod \"node-resolver-t9ks5\" (UID: \"690b430b-1984-4119-beae-85786c10b267\") " pod="openshift-dns/node-resolver-t9ks5"
Sep 18 13:58:51.916854 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.916565    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4626a667-4cd6-468c-8ef3-6e3a925c2608-systemd-units\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:51.916854 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.916594    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4626a667-4cd6-468c-8ef3-6e3a925c2608-run-systemd\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:51.916854 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.916616    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4626a667-4cd6-468c-8ef3-6e3a925c2608-host-cni-netd\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:51.916854 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.916637    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4626a667-4cd6-468c-8ef3-6e3a925c2608-ovnkube-config\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:51.916854 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.916654    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 18 13:58:51.916854 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.916661    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 18 13:58:51.916854 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.916681    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dbb463f3-8dcd-413d-859f-5dcf87c59115-host\") pod \"node-ca-8n79p\" (UID: \"dbb463f3-8dcd-413d-859f-5dcf87c59115\") " pod="openshift-image-registry/node-ca-8n79p"
Sep 18 13:58:51.916854 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.916723    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4626a667-4cd6-468c-8ef3-6e3a925c2608-host-run-netns\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:51.916854 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.916759    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4626a667-4cd6-468c-8ef3-6e3a925c2608-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:51.916854 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.916798    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4626a667-4cd6-468c-8ef3-6e3a925c2608-env-overrides\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:51.916854 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.916848    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dbb463f3-8dcd-413d-859f-5dcf87c59115-serviceca\") pod \"node-ca-8n79p\" (UID: \"dbb463f3-8dcd-413d-859f-5dcf87c59115\") " pod="openshift-image-registry/node-ca-8n79p"
Sep 18 13:58:51.917547 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.916883    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-h4j87\""
Sep 18 13:58:51.917547 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.916891    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57282ff1-fab4-4cd4-8087-5517bd249f10-metrics-certs\") pod \"network-metrics-daemon-2mgfl\" (UID: \"57282ff1-fab4-4cd4-8087-5517bd249f10\") " pod="openshift-multus/network-metrics-daemon-2mgfl"
Sep 18 13:58:51.917547 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.916931    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 18 13:58:51.917547 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.916939    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 18 13:58:51.917547 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.917166    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bthf7"
Sep 18 13:58:51.917547 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.917295    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 18 13:58:51.919544 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.919527    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2hvmm"
Sep 18 13:58:51.919665 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:51.919601    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2hvmm" podUID="ca72836e-7ed0-47e5-a106-2f47340eb6b1"
Sep 18 13:58:51.919949 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.919932    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 18 13:58:51.921540 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.920855    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-v6h9z\""
Sep 18 13:58:51.921540 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.921071    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 18 13:58:51.922125 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.922108    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fkrgg"
Sep 18 13:58:51.925416 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.925394    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 13:58:51.925579 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.925557    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-rd465\""
Sep 18 13:58:51.926526 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.925851    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-p5f5h"
Sep 18 13:58:51.926526 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.926000    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 18 13:58:51.928978 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.928240    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 13:58:51.929747 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.929729    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 18 13:58:51.930146 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.930131    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnrrz"
Sep 18 13:58:51.930444 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.930241    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zcwsc\""
Sep 18 13:58:51.931207 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.930960    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 18 13:58:51.932391 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.932305    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 18 13:58:51.932391 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.932340    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-d8w8d\""
Sep 18 13:58:51.932595 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.932577    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 18 13:58:51.933370 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.933342    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 18 13:58:51.948351 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.948335    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5zvgk"
Sep 18 13:58:51.956731 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:51.956523    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5zvgk"
Sep 18 13:58:52.007281 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.007256    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 18 13:58:52.017908 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.017884    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ca72836e-7ed0-47e5-a106-2f47340eb6b1-original-pull-secret\") pod \"global-pull-secret-syncer-2hvmm\" (UID: \"ca72836e-7ed0-47e5-a106-2f47340eb6b1\") " pod="kube-system/global-pull-secret-syncer-2hvmm"
Sep 18 13:58:52.018014 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.017921    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2201342-d012-4f49-aef5-4ca6901824d1-var-lib-kubelet\") pod \"tuned-bthf7\" (UID: \"f2201342-d012-4f49-aef5-4ca6901824d1\") " pod="openshift-cluster-node-tuning-operator/tuned-bthf7"
Sep 18 13:58:52.018014 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.017949    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4626a667-4cd6-468c-8ef3-6e3a925c2608-env-overrides\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:52.018014 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.017972    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ca72836e-7ed0-47e5-a106-2f47340eb6b1-kubelet-config\") pod \"global-pull-secret-syncer-2hvmm\" (UID: \"ca72836e-7ed0-47e5-a106-2f47340eb6b1\") " pod="kube-system/global-pull-secret-syncer-2hvmm"
Sep 18 13:58:52.018014 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.017995    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f2201342-d012-4f49-aef5-4ca6901824d1-etc-sysctl-conf\") pod \"tuned-bthf7\" (UID: \"f2201342-d012-4f49-aef5-4ca6901824d1\") " pod="openshift-cluster-node-tuning-operator/tuned-bthf7"
Sep 18 13:58:52.018237 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.018018    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2c6e66f3-7060-4ece-823b-1a90f55768b3-host-run-netns\") pod \"multus-czbmm\" (UID: \"2c6e66f3-7060-4ece-823b-1a90f55768b3\") " pod="openshift-multus/multus-czbmm"
Sep 18 13:58:52.018237 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.018055    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c6e66f3-7060-4ece-823b-1a90f55768b3-host-var-lib-kubelet\") pod \"multus-czbmm\" (UID: \"2c6e66f3-7060-4ece-823b-1a90f55768b3\") " pod="openshift-multus/multus-czbmm"
Sep 18 13:58:52.018237 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.018079    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f2201342-d012-4f49-aef5-4ca6901824d1-etc-sysctl-d\") pod \"tuned-bthf7\" (UID: \"f2201342-d012-4f49-aef5-4ca6901824d1\") " pod="openshift-cluster-node-tuning-operator/tuned-bthf7"
Sep 18 13:58:52.018237 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.018101    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f2201342-d012-4f49-aef5-4ca6901824d1-etc-systemd\") pod \"tuned-bthf7\" (UID: \"f2201342-d012-4f49-aef5-4ca6901824d1\") " pod="openshift-cluster-node-tuning-operator/tuned-bthf7"
Sep 18 13:58:52.018237 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.018123    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1375ec1e-4106-47ba-867a-42a96ed1014c-registration-dir\") pod \"aws-ebs-csi-driver-node-gnrrz\" (UID: \"1375ec1e-4106-47ba-867a-42a96ed1014c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnrrz"
Sep 18 13:58:52.018237 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.018149    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfq4k\" (UniqueName: \"kubernetes.io/projected/dbb463f3-8dcd-413d-859f-5dcf87c59115-kube-api-access-mfq4k\") pod \"node-ca-8n79p\" (UID: \"dbb463f3-8dcd-413d-859f-5dcf87c59115\") " pod="openshift-image-registry/node-ca-8n79p"
Sep 18 13:58:52.018237 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.018190    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4626a667-4cd6-468c-8ef3-6e3a925c2608-host-slash\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:52.018237 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.018224    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58a315d7-c375-4d9d-9b8b-fe148b1329e1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p5f5h\" (UID: \"58a315d7-c375-4d9d-9b8b-fe148b1329e1\") " pod="openshift-multus/multus-additional-cni-plugins-p5f5h"
Sep 18 13:58:52.018602 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.018247    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/58a315d7-c375-4d9d-9b8b-fe148b1329e1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p5f5h\" (UID: \"58a315d7-c375-4d9d-9b8b-fe148b1329e1\") " pod="openshift-multus/multus-additional-cni-plugins-p5f5h"
Sep 18 13:58:52.018602 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.018271    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htmqx\" (UniqueName: \"kubernetes.io/projected/58a315d7-c375-4d9d-9b8b-fe148b1329e1-kube-api-access-htmqx\") pod \"multus-additional-cni-plugins-p5f5h\" (UID: \"58a315d7-c375-4d9d-9b8b-fe148b1329e1\") " pod="openshift-multus/multus-additional-cni-plugins-p5f5h"
Sep 18 13:58:52.018602 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.018299    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4626a667-4cd6-468c-8ef3-6e3a925c2608-host-slash\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:52.018602 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.018299    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/690b430b-1984-4119-beae-85786c10b267-tmp-dir\") pod \"node-resolver-t9ks5\" (UID: \"690b430b-1984-4119-beae-85786c10b267\") " pod="openshift-dns/node-resolver-t9ks5"
Sep 18 13:58:52.018602 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.018349    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4626a667-4cd6-468c-8ef3-6e3a925c2608-run-openvswitch\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:52.018602 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.018372    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c6e66f3-7060-4ece-823b-1a90f55768b3-multus-cni-dir\") pod \"multus-czbmm\" (UID: \"2c6e66f3-7060-4ece-823b-1a90f55768b3\") " pod="openshift-multus/multus-czbmm"
Sep 18 13:58:52.018602 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.018387    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2c6e66f3-7060-4ece-823b-1a90f55768b3-multus-socket-dir-parent\") pod \"multus-czbmm\" (UID: \"2c6e66f3-7060-4ece-823b-1a90f55768b3\") " pod="openshift-multus/multus-czbmm"
Sep 18 13:58:52.018602 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.018412    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2c6e66f3-7060-4ece-823b-1a90f55768b3-hostroot\") pod \"multus-czbmm\" (UID: \"2c6e66f3-7060-4ece-823b-1a90f55768b3\") " pod="openshift-multus/multus-czbmm"
Sep 18 13:58:52.018602 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.018429    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1375ec1e-4106-47ba-867a-42a96ed1014c-sys-fs\") pod \"aws-ebs-csi-driver-node-gnrrz\" (UID: \"1375ec1e-4106-47ba-867a-42a96ed1014c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnrrz"
Sep 18 13:58:52.018602 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.018433    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4626a667-4cd6-468c-8ef3-6e3a925c2608-run-openvswitch\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:52.018602 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.018444    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kqlh\" (UniqueName: \"kubernetes.io/projected/1375ec1e-4106-47ba-867a-42a96ed1014c-kube-api-access-5kqlh\") pod \"aws-ebs-csi-driver-node-gnrrz\" (UID: \"1375ec1e-4106-47ba-867a-42a96ed1014c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnrrz"
Sep 18 13:58:52.018602 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.018440    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4626a667-4cd6-468c-8ef3-6e3a925c2608-env-overrides\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:52.018602 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.018474    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4626a667-4cd6-468c-8ef3-6e3a925c2608-ovn-node-metrics-cert\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:52.018602 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.018519    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4626a667-4cd6-468c-8ef3-6e3a925c2608-ovnkube-script-lib\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:52.018602 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.018567    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/690b430b-1984-4119-beae-85786c10b267-tmp-dir\") pod \"node-resolver-t9ks5\" (UID: \"690b430b-1984-4119-beae-85786c10b267\") " pod="openshift-dns/node-resolver-t9ks5"
Sep 18 13:58:52.018602 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.018590    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbd24\" (UniqueName: \"kubernetes.io/projected/57282ff1-fab4-4cd4-8087-5517bd249f10-kube-api-access-dbd24\") pod \"network-metrics-daemon-2mgfl\" (UID: \"57282ff1-fab4-4cd4-8087-5517bd249f10\") " pod="openshift-multus/network-metrics-daemon-2mgfl"
Sep 18 13:58:52.019358 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.018702    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2c6e66f3-7060-4ece-823b-1a90f55768b3-host-run-k8s-cni-cncf-io\") pod \"multus-czbmm\" (UID: \"2c6e66f3-7060-4ece-823b-1a90f55768b3\") " pod="openshift-multus/multus-czbmm"
Sep 18 13:58:52.019358 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.018737    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2c6e66f3-7060-4ece-823b-1a90f55768b3-multus-daemon-config\") pod \"multus-czbmm\" (UID: \"2c6e66f3-7060-4ece-823b-1a90f55768b3\") " pod="openshift-multus/multus-czbmm"
Sep 18 13:58:52.019358 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.018767    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7f65deed-0197-489a-a602-062486b50a57-iptables-alerter-script\") pod \"iptables-alerter-469c7\" (UID: \"7f65deed-0197-489a-a602-062486b50a57\") " pod="openshift-network-operator/iptables-alerter-469c7"
Sep 18 13:58:52.019358 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.018788    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/58a315d7-c375-4d9d-9b8b-fe148b1329e1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p5f5h\" (UID: \"58a315d7-c375-4d9d-9b8b-fe148b1329e1\") " pod="openshift-multus/multus-additional-cni-plugins-p5f5h"
Sep 18 13:58:52.019358 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.018811    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4626a667-4cd6-468c-8ef3-6e3a925c2608-var-lib-openvswitch\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:52.019358 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.018831    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4626a667-4cd6-468c-8ef3-6e3a925c2608-node-log\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:52.019358 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.018886    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4626a667-4cd6-468c-8ef3-6e3a925c2608-var-lib-openvswitch\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:52.019358 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.018910    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c6e66f3-7060-4ece-823b-1a90f55768b3-system-cni-dir\") pod \"multus-czbmm\" (UID: \"2c6e66f3-7060-4ece-823b-1a90f55768b3\") " pod="openshift-multus/multus-czbmm"
Sep 18 13:58:52.019358 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.018929    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcnx9\" (UniqueName: \"kubernetes.io/projected/2c6e66f3-7060-4ece-823b-1a90f55768b3-kube-api-access-tcnx9\") pod \"multus-czbmm\" (UID: \"2c6e66f3-7060-4ece-823b-1a90f55768b3\") " pod="openshift-multus/multus-czbmm"
Sep 18 13:58:52.019358 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.018953    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm9qk\" (UniqueName: \"kubernetes.io/projected/7f65deed-0197-489a-a602-062486b50a57-kube-api-access-xm9qk\") pod \"iptables-alerter-469c7\" (UID: \"7f65deed-0197-489a-a602-062486b50a57\") " pod="openshift-network-operator/iptables-alerter-469c7"
Sep 18 13:58:52.019358 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.018975    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f2201342-d012-4f49-aef5-4ca6901824d1-run\") pod \"tuned-bthf7\" (UID: \"f2201342-d012-4f49-aef5-4ca6901824d1\") " pod="openshift-cluster-node-tuning-operator/tuned-bthf7"
Sep 18 13:58:52.019358 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.018992    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f2201342-d012-4f49-aef5-4ca6901824d1-host\") pod \"tuned-bthf7\" (UID: \"f2201342-d012-4f49-aef5-4ca6901824d1\") " pod="openshift-cluster-node-tuning-operator/tuned-bthf7"
Sep 18 13:58:52.019358 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.019016    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zndv4\" (UniqueName: \"kubernetes.io/projected/690b430b-1984-4119-beae-85786c10b267-kube-api-access-zndv4\") pod \"node-resolver-t9ks5\" (UID: \"690b430b-1984-4119-beae-85786c10b267\") " pod="openshift-dns/node-resolver-t9ks5"
Sep 18 13:58:52.019358 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.019029    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4626a667-4cd6-468c-8ef3-6e3a925c2608-ovnkube-script-lib\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:52.019358 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.019039    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4626a667-4cd6-468c-8ef3-6e3a925c2608-systemd-units\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:52.019358 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.019078    2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 18 13:58:52.019358 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.019120    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4626a667-4cd6-468c-8ef3-6e3a925c2608-node-log\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:52.020146 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.019157    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4626a667-4cd6-468c-8ef3-6e3a925c2608-systemd-units\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:52.020146 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.019351    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4626a667-4cd6-468c-8ef3-6e3a925c2608-run-systemd\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:52.020146 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.019379    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4626a667-4cd6-468c-8ef3-6e3a925c2608-host-cni-netd\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:52.020146 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.019404    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4626a667-4cd6-468c-8ef3-6e3a925c2608-ovnkube-config\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:52.020146 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.019412    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4626a667-4cd6-468c-8ef3-6e3a925c2608-run-systemd\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:52.020146 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.019428    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f2201342-d012-4f49-aef5-4ca6901824d1-lib-modules\") pod \"tuned-bthf7\" (UID: \"f2201342-d012-4f49-aef5-4ca6901824d1\") " pod="openshift-cluster-node-tuning-operator/tuned-bthf7"
Sep 18 13:58:52.020146 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.019454    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl6x2\" (UniqueName: \"kubernetes.io/projected/f2201342-d012-4f49-aef5-4ca6901824d1-kube-api-access-vl6x2\") pod \"tuned-bthf7\" (UID: \"f2201342-d012-4f49-aef5-4ca6901824d1\") " pod="openshift-cluster-node-tuning-operator/tuned-bthf7"
Sep 18 13:58:52.020146 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.019457    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4626a667-4cd6-468c-8ef3-6e3a925c2608-host-cni-netd\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:52.020146 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.019486    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2c6e66f3-7060-4ece-823b-1a90f55768b3-host-var-lib-cni-multus\") pod \"multus-czbmm\" (UID: \"2c6e66f3-7060-4ece-823b-1a90f55768b3\") " pod="openshift-multus/multus-czbmm"
Sep 18 13:58:52.020146 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.019511    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9f51a32f-f091-49b2-97f8-7e3ba0008474-konnectivity-ca\") pod \"konnectivity-agent-fkrgg\" (UID: \"9f51a32f-f091-49b2-97f8-7e3ba0008474\") " pod="kube-system/konnectivity-agent-fkrgg"
Sep 18 13:58:52.020146 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.019534    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1375ec1e-4106-47ba-867a-42a96ed1014c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gnrrz\" (UID: \"1375ec1e-4106-47ba-867a-42a96ed1014c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnrrz"
Sep 18 13:58:52.020146 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.019557    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1375ec1e-4106-47ba-867a-42a96ed1014c-etc-selinux\") pod \"aws-ebs-csi-driver-node-gnrrz\" (UID: \"1375ec1e-4106-47ba-867a-42a96ed1014c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnrrz"
Sep 18 13:58:52.020146 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.019603    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dbb463f3-8dcd-413d-859f-5dcf87c59115-host\") pod \"node-ca-8n79p\" (UID: \"dbb463f3-8dcd-413d-859f-5dcf87c59115\") " pod="openshift-image-registry/node-ca-8n79p"
Sep 18 13:58:52.020146 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.019634    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4626a667-4cd6-468c-8ef3-6e3a925c2608-host-run-netns\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:52.020146 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.019656    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4626a667-4cd6-468c-8ef3-6e3a925c2608-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:52.020146 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.019672    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c6e66f3-7060-4ece-823b-1a90f55768b3-multus-conf-dir\") pod \"multus-czbmm\" (UID: \"2c6e66f3-7060-4ece-823b-1a90f55768b3\") " pod="openshift-multus/multus-czbmm"
Sep 18 13:58:52.020146 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.019670    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dbb463f3-8dcd-413d-859f-5dcf87c59115-host\") pod \"node-ca-8n79p\" (UID: \"dbb463f3-8dcd-413d-859f-5dcf87c59115\") " pod="openshift-image-registry/node-ca-8n79p"
Sep 18 13:58:52.021000 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.019685    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9f51a32f-f091-49b2-97f8-7e3ba0008474-agent-certs\") pod \"konnectivity-agent-fkrgg\" (UID: \"9f51a32f-f091-49b2-97f8-7e3ba0008474\") " pod="kube-system/konnectivity-agent-fkrgg"
Sep 18 13:58:52.021000 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.019700    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f2201342-d012-4f49-aef5-4ca6901824d1-etc-sysconfig\") pod \"tuned-bthf7\" (UID: \"f2201342-d012-4f49-aef5-4ca6901824d1\") " pod="openshift-cluster-node-tuning-operator/tuned-bthf7"
Sep 18 13:58:52.021000 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.019712    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4626a667-4cd6-468c-8ef3-6e3a925c2608-host-run-netns\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:52.021000 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.019722    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2201342-d012-4f49-aef5-4ca6901824d1-etc-kubernetes\") pod \"tuned-bthf7\" (UID: \"f2201342-d012-4f49-aef5-4ca6901824d1\") " pod="openshift-cluster-node-tuning-operator/tuned-bthf7"
Sep 18 13:58:52.021000 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.019727    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4626a667-4cd6-468c-8ef3-6e3a925c2608-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:52.021000 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.019754    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dbb463f3-8dcd-413d-859f-5dcf87c59115-serviceca\") pod \"node-ca-8n79p\" (UID: \"dbb463f3-8dcd-413d-859f-5dcf87c59115\") " pod="openshift-image-registry/node-ca-8n79p"
Sep 18 13:58:52.021000 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.019803    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57282ff1-fab4-4cd4-8087-5517bd249f10-metrics-certs\") pod \"network-metrics-daemon-2mgfl\" (UID: \"57282ff1-fab4-4cd4-8087-5517bd249f10\") " pod="openshift-multus/network-metrics-daemon-2mgfl"
Sep 18 13:58:52.021000 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.019844    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2c6e66f3-7060-4ece-823b-1a90f55768b3-cni-binary-copy\") pod \"multus-czbmm\" (UID: \"2c6e66f3-7060-4ece-823b-1a90f55768b3\") " pod="openshift-multus/multus-czbmm"
Sep 18 13:58:52.021000 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.019887    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2c6e66f3-7060-4ece-823b-1a90f55768b3-host-var-lib-cni-bin\") pod \"multus-czbmm\" (UID: \"2c6e66f3-7060-4ece-823b-1a90f55768b3\") " pod="openshift-multus/multus-czbmm"
Sep 18 13:58:52.021000 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.019911    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c6e66f3-7060-4ece-823b-1a90f55768b3-etc-kubernetes\") pod \"multus-czbmm\" (UID: \"2c6e66f3-7060-4ece-823b-1a90f55768b3\") " pod="openshift-multus/multus-czbmm"
Sep 18 13:58:52.021000 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.019912    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4626a667-4cd6-468c-8ef3-6e3a925c2608-ovnkube-config\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:52.021000 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:52.019920    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:58:52.021000 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.019935    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1375ec1e-4106-47ba-867a-42a96ed1014c-device-dir\") pod \"aws-ebs-csi-driver-node-gnrrz\" (UID: \"1375ec1e-4106-47ba-867a-42a96ed1014c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnrrz"
Sep 18 13:58:52.021000 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.019962    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/690b430b-1984-4119-beae-85786c10b267-hosts-file\") pod \"node-resolver-t9ks5\" (UID: \"690b430b-1984-4119-beae-85786c10b267\") " pod="openshift-dns/node-resolver-t9ks5"
Sep 18 13:58:52.021000 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:52.019981    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57282ff1-fab4-4cd4-8087-5517bd249f10-metrics-certs podName:57282ff1-fab4-4cd4-8087-5517bd249f10 nodeName:}" failed. No retries permitted until 2026-09-18 13:58:52.519962075 +0000 UTC m=+4.408678008 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57282ff1-fab4-4cd4-8087-5517bd249f10-metrics-certs") pod "network-metrics-daemon-2mgfl" (UID: "57282ff1-fab4-4cd4-8087-5517bd249f10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:58:52.021000 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.020010    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4626a667-4cd6-468c-8ef3-6e3a925c2608-etc-openvswitch\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:52.021000 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.020019    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/690b430b-1984-4119-beae-85786c10b267-hosts-file\") pod \"node-resolver-t9ks5\" (UID: \"690b430b-1984-4119-beae-85786c10b267\") " pod="openshift-dns/node-resolver-t9ks5"
Sep 18 13:58:52.021858 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.020033    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r7hs\" (UniqueName: \"kubernetes.io/projected/4626a667-4cd6-468c-8ef3-6e3a925c2608-kube-api-access-7r7hs\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:52.021858 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.020055    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c6e66f3-7060-4ece-823b-1a90f55768b3-cnibin\") pod \"multus-czbmm\" (UID: \"2c6e66f3-7060-4ece-823b-1a90f55768b3\") " pod="openshift-multus/multus-czbmm"
Sep 18 13:58:52.021858 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.020062    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4626a667-4cd6-468c-8ef3-6e3a925c2608-etc-openvswitch\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:52.021858 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.020075    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4626a667-4cd6-468c-8ef3-6e3a925c2608-host-kubelet\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:52.021858 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.020098    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4626a667-4cd6-468c-8ef3-6e3a925c2608-host-run-ovn-kubernetes\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:52.021858 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.020116    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2201342-d012-4f49-aef5-4ca6901824d1-sys\") pod \"tuned-bthf7\" (UID: \"f2201342-d012-4f49-aef5-4ca6901824d1\") " pod="openshift-cluster-node-tuning-operator/tuned-bthf7"
Sep 18 13:58:52.021858 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.020130    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4626a667-4cd6-468c-8ef3-6e3a925c2608-host-kubelet\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:52.021858 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.020144    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58a315d7-c375-4d9d-9b8b-fe148b1329e1-system-cni-dir\") pod \"multus-additional-cni-plugins-p5f5h\" (UID: \"58a315d7-c375-4d9d-9b8b-fe148b1329e1\") " pod="openshift-multus/multus-additional-cni-plugins-p5f5h"
Sep 18 13:58:52.021858 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.020157    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dbb463f3-8dcd-413d-859f-5dcf87c59115-serviceca\") pod \"node-ca-8n79p\" (UID: \"dbb463f3-8dcd-413d-859f-5dcf87c59115\") " pod="openshift-image-registry/node-ca-8n79p"
Sep 18 13:58:52.021858 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.020163    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4626a667-4cd6-468c-8ef3-6e3a925c2608-host-run-ovn-kubernetes\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:52.021858 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.020166    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58a315d7-c375-4d9d-9b8b-fe148b1329e1-cnibin\") pod \"multus-additional-cni-plugins-p5f5h\" (UID: \"58a315d7-c375-4d9d-9b8b-fe148b1329e1\") " pod="openshift-multus/multus-additional-cni-plugins-p5f5h"
Sep 18 13:58:52.021858 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.020256    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4626a667-4cd6-468c-8ef3-6e3a925c2608-run-ovn\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:52.021858 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.020290    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4626a667-4cd6-468c-8ef3-6e3a925c2608-host-cni-bin\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:52.021858 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.020302    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4626a667-4cd6-468c-8ef3-6e3a925c2608-run-ovn\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:52.021858 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.020334    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ca72836e-7ed0-47e5-a106-2f47340eb6b1-dbus\") pod \"global-pull-secret-syncer-2hvmm\" (UID: \"ca72836e-7ed0-47e5-a106-2f47340eb6b1\") " pod="kube-system/global-pull-secret-syncer-2hvmm"
Sep 18 13:58:52.021858 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.020353    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4626a667-4cd6-468c-8ef3-6e3a925c2608-host-cni-bin\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:52.021858 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.020359    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7f65deed-0197-489a-a602-062486b50a57-host-slash\") pod \"iptables-alerter-469c7\" (UID: \"7f65deed-0197-489a-a602-062486b50a57\") " pod="openshift-network-operator/iptables-alerter-469c7"
Sep 18 13:58:52.022721 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.020398    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f2201342-d012-4f49-aef5-4ca6901824d1-etc-modprobe-d\") pod \"tuned-bthf7\" (UID: \"f2201342-d012-4f49-aef5-4ca6901824d1\") " pod="openshift-cluster-node-tuning-operator/tuned-bthf7"
Sep 18 13:58:52.022721 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.020429    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2201342-d012-4f49-aef5-4ca6901824d1-tmp\") pod \"tuned-bthf7\" (UID: \"f2201342-d012-4f49-aef5-4ca6901824d1\") " pod="openshift-cluster-node-tuning-operator/tuned-bthf7"
Sep 18 13:58:52.022721 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.020481    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1375ec1e-4106-47ba-867a-42a96ed1014c-socket-dir\") pod \"aws-ebs-csi-driver-node-gnrrz\" (UID: \"1375ec1e-4106-47ba-867a-42a96ed1014c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnrrz"
Sep 18 13:58:52.022721 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.020519    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4626a667-4cd6-468c-8ef3-6e3a925c2608-log-socket\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:52.022721 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.020557    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4626a667-4cd6-468c-8ef3-6e3a925c2608-log-socket\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:52.022721 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.020561    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t8sk\" (UniqueName: \"kubernetes.io/projected/7e8de495-7804-4c38-af57-655459217f1c-kube-api-access-4t8sk\") pod \"network-check-target-j7msq\" (UID: \"7e8de495-7804-4c38-af57-655459217f1c\") " pod="openshift-network-diagnostics/network-check-target-j7msq"
Sep 18 13:58:52.022721 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.020677    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/58a315d7-c375-4d9d-9b8b-fe148b1329e1-cni-binary-copy\") pod \"multus-additional-cni-plugins-p5f5h\" (UID: \"58a315d7-c375-4d9d-9b8b-fe148b1329e1\") " pod="openshift-multus/multus-additional-cni-plugins-p5f5h"
Sep 18 13:58:52.022721 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.020706    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f2201342-d012-4f49-aef5-4ca6901824d1-etc-tuned\") pod \"tuned-bthf7\" (UID: \"f2201342-d012-4f49-aef5-4ca6901824d1\") " pod="openshift-cluster-node-tuning-operator/tuned-bthf7"
Sep 18 13:58:52.022721 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.020728    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2c6e66f3-7060-4ece-823b-1a90f55768b3-host-run-multus-certs\") pod \"multus-czbmm\" (UID: \"2c6e66f3-7060-4ece-823b-1a90f55768b3\") " pod="openshift-multus/multus-czbmm"
Sep 18 13:58:52.023126 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.022881    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4626a667-4cd6-468c-8ef3-6e3a925c2608-ovn-node-metrics-cert\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:52.026722 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.026689    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfq4k\" (UniqueName: \"kubernetes.io/projected/dbb463f3-8dcd-413d-859f-5dcf87c59115-kube-api-access-mfq4k\") pod \"node-ca-8n79p\" (UID: \"dbb463f3-8dcd-413d-859f-5dcf87c59115\") " pod="openshift-image-registry/node-ca-8n79p"
Sep 18 13:58:52.027727 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.027707    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zndv4\" (UniqueName: \"kubernetes.io/projected/690b430b-1984-4119-beae-85786c10b267-kube-api-access-zndv4\") pod \"node-resolver-t9ks5\" (UID: \"690b430b-1984-4119-beae-85786c10b267\") " pod="openshift-dns/node-resolver-t9ks5"
Sep 18 13:58:52.028152 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.028134    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbd24\" (UniqueName: \"kubernetes.io/projected/57282ff1-fab4-4cd4-8087-5517bd249f10-kube-api-access-dbd24\") pod \"network-metrics-daemon-2mgfl\" (UID: \"57282ff1-fab4-4cd4-8087-5517bd249f10\") " pod="openshift-multus/network-metrics-daemon-2mgfl"
Sep 18 13:58:52.035390 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:52.035368    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 13:58:52.035486 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:52.035396    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 13:58:52.035486 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:52.035409    2579 projected.go:196] Error preparing data for projected volume kube-api-access-4t8sk for pod openshift-network-diagnostics/network-check-target-j7msq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:58:52.035589 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:52.035501    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7e8de495-7804-4c38-af57-655459217f1c-kube-api-access-4t8sk podName:7e8de495-7804-4c38-af57-655459217f1c nodeName:}" failed. No retries permitted until 2026-09-18 13:58:52.535484814 +0000 UTC m=+4.424200733 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4t8sk" (UniqueName: "kubernetes.io/projected/7e8de495-7804-4c38-af57-655459217f1c-kube-api-access-4t8sk") pod "network-check-target-j7msq" (UID: "7e8de495-7804-4c38-af57-655459217f1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:58:52.035660 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.035609    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r7hs\" (UniqueName: \"kubernetes.io/projected/4626a667-4cd6-468c-8ef3-6e3a925c2608-kube-api-access-7r7hs\") pod \"ovnkube-node-sg4tv\" (UID: \"4626a667-4cd6-468c-8ef3-6e3a925c2608\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:52.121137 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.121114    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2c6e66f3-7060-4ece-823b-1a90f55768b3-multus-socket-dir-parent\") pod \"multus-czbmm\" (UID: \"2c6e66f3-7060-4ece-823b-1a90f55768b3\") " pod="openshift-multus/multus-czbmm"
Sep 18 13:58:52.121240 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.121149    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2c6e66f3-7060-4ece-823b-1a90f55768b3-hostroot\") pod \"multus-czbmm\" (UID: \"2c6e66f3-7060-4ece-823b-1a90f55768b3\") " pod="openshift-multus/multus-czbmm"
Sep 18 13:58:52.121240 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.121187    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1375ec1e-4106-47ba-867a-42a96ed1014c-sys-fs\") pod \"aws-ebs-csi-driver-node-gnrrz\" (UID: \"1375ec1e-4106-47ba-867a-42a96ed1014c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnrrz"
Sep 18 13:58:52.121240 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.121208    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kqlh\" (UniqueName: \"kubernetes.io/projected/1375ec1e-4106-47ba-867a-42a96ed1014c-kube-api-access-5kqlh\") pod \"aws-ebs-csi-driver-node-gnrrz\" (UID: \"1375ec1e-4106-47ba-867a-42a96ed1014c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnrrz"
Sep 18 13:58:52.121240 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.121225    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2c6e66f3-7060-4ece-823b-1a90f55768b3-host-run-k8s-cni-cncf-io\") pod \"multus-czbmm\" (UID: \"2c6e66f3-7060-4ece-823b-1a90f55768b3\") " pod="openshift-multus/multus-czbmm"
Sep 18 13:58:52.121240 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.121238    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2c6e66f3-7060-4ece-823b-1a90f55768b3-multus-daemon-config\") pod \"multus-czbmm\" (UID: \"2c6e66f3-7060-4ece-823b-1a90f55768b3\") " pod="openshift-multus/multus-czbmm"
Sep 18 13:58:52.121483 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.121251    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7f65deed-0197-489a-a602-062486b50a57-iptables-alerter-script\") pod \"iptables-alerter-469c7\" (UID: \"7f65deed-0197-489a-a602-062486b50a57\") " pod="openshift-network-operator/iptables-alerter-469c7"
Sep 18 13:58:52.121483 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.121262    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1375ec1e-4106-47ba-867a-42a96ed1014c-sys-fs\") pod \"aws-ebs-csi-driver-node-gnrrz\" (UID: \"1375ec1e-4106-47ba-867a-42a96ed1014c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnrrz"
Sep 18 13:58:52.121483 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.121272    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/58a315d7-c375-4d9d-9b8b-fe148b1329e1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p5f5h\" (UID: \"58a315d7-c375-4d9d-9b8b-fe148b1329e1\") " pod="openshift-multus/multus-additional-cni-plugins-p5f5h"
Sep 18 13:58:52.121483 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.121292    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2c6e66f3-7060-4ece-823b-1a90f55768b3-host-run-k8s-cni-cncf-io\") pod \"multus-czbmm\" (UID: \"2c6e66f3-7060-4ece-823b-1a90f55768b3\") " pod="openshift-multus/multus-czbmm"
Sep 18 13:58:52.121483 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.121302    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c6e66f3-7060-4ece-823b-1a90f55768b3-system-cni-dir\") pod \"multus-czbmm\" (UID: \"2c6e66f3-7060-4ece-823b-1a90f55768b3\") " pod="openshift-multus/multus-czbmm"
Sep 18 13:58:52.121483 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.121298    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2c6e66f3-7060-4ece-823b-1a90f55768b3-multus-socket-dir-parent\") pod \"multus-czbmm\" (UID: \"2c6e66f3-7060-4ece-823b-1a90f55768b3\") " pod="openshift-multus/multus-czbmm"
Sep 18 13:58:52.121483 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.121320    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcnx9\" (UniqueName: \"kubernetes.io/projected/2c6e66f3-7060-4ece-823b-1a90f55768b3-kube-api-access-tcnx9\") pod \"multus-czbmm\" (UID: \"2c6e66f3-7060-4ece-823b-1a90f55768b3\") " pod="openshift-multus/multus-czbmm"
Sep 18 13:58:52.121483 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.121306    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2c6e66f3-7060-4ece-823b-1a90f55768b3-hostroot\") pod \"multus-czbmm\" (UID: \"2c6e66f3-7060-4ece-823b-1a90f55768b3\") " pod="openshift-multus/multus-czbmm"
Sep 18 13:58:52.121483 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.121416    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c6e66f3-7060-4ece-823b-1a90f55768b3-system-cni-dir\") pod \"multus-czbmm\" (UID: \"2c6e66f3-7060-4ece-823b-1a90f55768b3\") " pod="openshift-multus/multus-czbmm"
Sep 18 13:58:52.121859 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.121506    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm9qk\" (UniqueName: \"kubernetes.io/projected/7f65deed-0197-489a-a602-062486b50a57-kube-api-access-xm9qk\") pod \"iptables-alerter-469c7\" (UID: \"7f65deed-0197-489a-a602-062486b50a57\") " pod="openshift-network-operator/iptables-alerter-469c7"
Sep 18 13:58:52.121859 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.121536    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f2201342-d012-4f49-aef5-4ca6901824d1-run\") pod \"tuned-bthf7\" (UID: \"f2201342-d012-4f49-aef5-4ca6901824d1\") " pod="openshift-cluster-node-tuning-operator/tuned-bthf7"
Sep 18 13:58:52.121859 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.121559    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f2201342-d012-4f49-aef5-4ca6901824d1-host\") pod \"tuned-bthf7\" (UID: \"f2201342-d012-4f49-aef5-4ca6901824d1\") " pod="openshift-cluster-node-tuning-operator/tuned-bthf7"
Sep 18 13:58:52.121859 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.121588    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f2201342-d012-4f49-aef5-4ca6901824d1-lib-modules\") pod \"tuned-bthf7\" (UID: \"f2201342-d012-4f49-aef5-4ca6901824d1\") " pod="openshift-cluster-node-tuning-operator/tuned-bthf7"
Sep 18 13:58:52.121859 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.121611    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl6x2\" (UniqueName: \"kubernetes.io/projected/f2201342-d012-4f49-aef5-4ca6901824d1-kube-api-access-vl6x2\") pod \"tuned-bthf7\" (UID: \"f2201342-d012-4f49-aef5-4ca6901824d1\") " pod="openshift-cluster-node-tuning-operator/tuned-bthf7"
Sep 18 13:58:52.121859 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.121635    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2c6e66f3-7060-4ece-823b-1a90f55768b3-host-var-lib-cni-multus\") pod \"multus-czbmm\" (UID: \"2c6e66f3-7060-4ece-823b-1a90f55768b3\") " pod="openshift-multus/multus-czbmm"
Sep 18 13:58:52.121859 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.121661    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9f51a32f-f091-49b2-97f8-7e3ba0008474-konnectivity-ca\") pod \"konnectivity-agent-fkrgg\" (UID: \"9f51a32f-f091-49b2-97f8-7e3ba0008474\") " pod="kube-system/konnectivity-agent-fkrgg"
Sep 18 13:58:52.121859 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.121685    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1375ec1e-4106-47ba-867a-42a96ed1014c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gnrrz\" (UID: \"1375ec1e-4106-47ba-867a-42a96ed1014c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnrrz"
Sep 18 13:58:52.121859 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.121708    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1375ec1e-4106-47ba-867a-42a96ed1014c-etc-selinux\") pod \"aws-ebs-csi-driver-node-gnrrz\" (UID: \"1375ec1e-4106-47ba-867a-42a96ed1014c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnrrz"
Sep 18 13:58:52.121859 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.121734    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c6e66f3-7060-4ece-823b-1a90f55768b3-multus-conf-dir\") pod \"multus-czbmm\" (UID: \"2c6e66f3-7060-4ece-823b-1a90f55768b3\") " pod="openshift-multus/multus-czbmm"
Sep 18 13:58:52.121859 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.121758    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9f51a32f-f091-49b2-97f8-7e3ba0008474-agent-certs\") pod \"konnectivity-agent-fkrgg\" (UID: \"9f51a32f-f091-49b2-97f8-7e3ba0008474\") " pod="kube-system/konnectivity-agent-fkrgg"
Sep 18 13:58:52.121859 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.121781    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f2201342-d012-4f49-aef5-4ca6901824d1-etc-sysconfig\") pod \"tuned-bthf7\" (UID: \"f2201342-d012-4f49-aef5-4ca6901824d1\") " pod="openshift-cluster-node-tuning-operator/tuned-bthf7"
Sep 18 13:58:52.121859 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.121805    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2201342-d012-4f49-aef5-4ca6901824d1-etc-kubernetes\") pod \"tuned-bthf7\" (UID: \"f2201342-d012-4f49-aef5-4ca6901824d1\") " pod="openshift-cluster-node-tuning-operator/tuned-bthf7"
Sep 18 13:58:52.121859 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.121836    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7f65deed-0197-489a-a602-062486b50a57-iptables-alerter-script\") pod \"iptables-alerter-469c7\" (UID: \"7f65deed-0197-489a-a602-062486b50a57\") " pod="openshift-network-operator/iptables-alerter-469c7"
Sep 18 13:58:52.121859 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.121843    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2c6e66f3-7060-4ece-823b-1a90f55768b3-cni-binary-copy\") pod \"multus-czbmm\" (UID: \"2c6e66f3-7060-4ece-823b-1a90f55768b3\") " pod="openshift-multus/multus-czbmm"
Sep 18 13:58:52.121859 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.121861    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/58a315d7-c375-4d9d-9b8b-fe148b1329e1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p5f5h\" (UID: \"58a315d7-c375-4d9d-9b8b-fe148b1329e1\") " pod="openshift-multus/multus-additional-cni-plugins-p5f5h"
Sep 18 13:58:52.122710 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.121869    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2c6e66f3-7060-4ece-823b-1a90f55768b3-host-var-lib-cni-bin\") pod \"multus-czbmm\" (UID: \"2c6e66f3-7060-4ece-823b-1a90f55768b3\") " pod="openshift-multus/multus-czbmm"
Sep 18 13:58:52.122710 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.121907    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2c6e66f3-7060-4ece-823b-1a90f55768b3-host-var-lib-cni-bin\") pod \"multus-czbmm\" (UID: \"2c6e66f3-7060-4ece-823b-1a90f55768b3\") " pod="openshift-multus/multus-czbmm"
Sep 18 13:58:52.122710 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.121916    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c6e66f3-7060-4ece-823b-1a90f55768b3-etc-kubernetes\") pod \"multus-czbmm\" (UID: \"2c6e66f3-7060-4ece-823b-1a90f55768b3\") " pod="openshift-multus/multus-czbmm"
Sep 18 13:58:52.122710 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.121962    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2c6e66f3-7060-4ece-823b-1a90f55768b3-host-var-lib-cni-multus\") pod \"multus-czbmm\" (UID: \"2c6e66f3-7060-4ece-823b-1a90f55768b3\") " pod="openshift-multus/multus-czbmm"
Sep 18 13:58:52.122710 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.121944    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1375ec1e-4106-47ba-867a-42a96ed1014c-device-dir\") pod \"aws-ebs-csi-driver-node-gnrrz\" (UID: \"1375ec1e-4106-47ba-867a-42a96ed1014c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnrrz"
Sep 18 13:58:52.122710 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.121977    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1375ec1e-4106-47ba-867a-42a96ed1014c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gnrrz\" (UID: \"1375ec1e-4106-47ba-867a-42a96ed1014c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnrrz"
Sep 18 13:58:52.122710 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.121996    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c6e66f3-7060-4ece-823b-1a90f55768b3-cnibin\") pod \"multus-czbmm\" (UID: \"2c6e66f3-7060-4ece-823b-1a90f55768b3\") " pod="openshift-multus/multus-czbmm"
Sep 18 13:58:52.122710 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122008    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c6e66f3-7060-4ece-823b-1a90f55768b3-etc-kubernetes\") pod \"multus-czbmm\" (UID: \"2c6e66f3-7060-4ece-823b-1a90f55768b3\") " pod="openshift-multus/multus-czbmm"
Sep 18 13:58:52.122710 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122024    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2201342-d012-4f49-aef5-4ca6901824d1-sys\") pod \"tuned-bthf7\" (UID: \"f2201342-d012-4f49-aef5-4ca6901824d1\") " pod="openshift-cluster-node-tuning-operator/tuned-bthf7"
Sep 18 13:58:52.122710 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122034    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f2201342-d012-4f49-aef5-4ca6901824d1-host\") pod \"tuned-bthf7\" (UID: \"f2201342-d012-4f49-aef5-4ca6901824d1\") " pod="openshift-cluster-node-tuning-operator/tuned-bthf7"
Sep 18 13:58:52.122710 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122049    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58a315d7-c375-4d9d-9b8b-fe148b1329e1-system-cni-dir\") pod \"multus-additional-cni-plugins-p5f5h\" (UID: \"58a315d7-c375-4d9d-9b8b-fe148b1329e1\") " pod="openshift-multus/multus-additional-cni-plugins-p5f5h"
Sep 18 13:58:52.122710 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122073    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58a315d7-c375-4d9d-9b8b-fe148b1329e1-cnibin\") pod \"multus-additional-cni-plugins-p5f5h\" (UID: \"58a315d7-c375-4d9d-9b8b-fe148b1329e1\") " pod="openshift-multus/multus-additional-cni-plugins-p5f5h"
Sep 18 13:58:52.122710 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122080    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f2201342-d012-4f49-aef5-4ca6901824d1-run\") pod \"tuned-bthf7\" (UID: \"f2201342-d012-4f49-aef5-4ca6901824d1\") " pod="openshift-cluster-node-tuning-operator/tuned-bthf7"
Sep 18 13:58:52.122710 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122100    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ca72836e-7ed0-47e5-a106-2f47340eb6b1-dbus\") pod \"global-pull-secret-syncer-2hvmm\" (UID: \"ca72836e-7ed0-47e5-a106-2f47340eb6b1\") " pod="kube-system/global-pull-secret-syncer-2hvmm"
Sep 18 13:58:52.122710 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122114    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2c6e66f3-7060-4ece-823b-1a90f55768b3-multus-daemon-config\") pod \"multus-czbmm\" (UID: \"2c6e66f3-7060-4ece-823b-1a90f55768b3\") " pod="openshift-multus/multus-czbmm"
Sep 18 13:58:52.122710 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122124    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c6e66f3-7060-4ece-823b-1a90f55768b3-multus-conf-dir\") pod \"multus-czbmm\" (UID: \"2c6e66f3-7060-4ece-823b-1a90f55768b3\") " pod="openshift-multus/multus-czbmm"
Sep 18 13:58:52.122710 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122125    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7f65deed-0197-489a-a602-062486b50a57-host-slash\") pod \"iptables-alerter-469c7\" (UID: \"7f65deed-0197-489a-a602-062486b50a57\") " pod="openshift-network-operator/iptables-alerter-469c7"
Sep 18 13:58:52.122710 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122163    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f2201342-d012-4f49-aef5-4ca6901824d1-etc-modprobe-d\") pod \"tuned-bthf7\" (UID: \"f2201342-d012-4f49-aef5-4ca6901824d1\") " pod="openshift-cluster-node-tuning-operator/tuned-bthf7"
Sep 18 13:58:52.123635 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122164    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1375ec1e-4106-47ba-867a-42a96ed1014c-device-dir\") pod \"aws-ebs-csi-driver-node-gnrrz\" (UID: \"1375ec1e-4106-47ba-867a-42a96ed1014c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnrrz"
Sep 18 13:58:52.123635 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122167    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7f65deed-0197-489a-a602-062486b50a57-host-slash\") pod \"iptables-alerter-469c7\" (UID: \"7f65deed-0197-489a-a602-062486b50a57\") " pod="openshift-network-operator/iptables-alerter-469c7"
Sep 18 13:58:52.123635 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122186    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58a315d7-c375-4d9d-9b8b-fe148b1329e1-cnibin\") pod \"multus-additional-cni-plugins-p5f5h\" (UID: \"58a315d7-c375-4d9d-9b8b-fe148b1329e1\") " pod="openshift-multus/multus-additional-cni-plugins-p5f5h"
Sep 18 13:58:52.123635 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.121905    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1375ec1e-4106-47ba-867a-42a96ed1014c-etc-selinux\") pod \"aws-ebs-csi-driver-node-gnrrz\" (UID: \"1375ec1e-4106-47ba-867a-42a96ed1014c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnrrz"
Sep 18 13:58:52.123635 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122207    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f2201342-d012-4f49-aef5-4ca6901824d1-lib-modules\") pod \"tuned-bthf7\" (UID: \"f2201342-d012-4f49-aef5-4ca6901824d1\") " pod="openshift-cluster-node-tuning-operator/tuned-bthf7"
Sep 18 13:58:52.123635 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122214    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2201342-d012-4f49-aef5-4ca6901824d1-tmp\") pod \"tuned-bthf7\" (UID: \"f2201342-d012-4f49-aef5-4ca6901824d1\") " pod="openshift-cluster-node-tuning-operator/tuned-bthf7"
Sep 18 13:58:52.123635 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122233    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58a315d7-c375-4d9d-9b8b-fe148b1329e1-system-cni-dir\") pod \"multus-additional-cni-plugins-p5f5h\" (UID: \"58a315d7-c375-4d9d-9b8b-fe148b1329e1\") " pod="openshift-multus/multus-additional-cni-plugins-p5f5h"
Sep 18 13:58:52.123635 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122253    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2201342-d012-4f49-aef5-4ca6901824d1-sys\") pod \"tuned-bthf7\" (UID: \"f2201342-d012-4f49-aef5-4ca6901824d1\") " pod="openshift-cluster-node-tuning-operator/tuned-bthf7"
Sep 18 13:58:52.123635 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122265    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1375ec1e-4106-47ba-867a-42a96ed1014c-socket-dir\") pod \"aws-ebs-csi-driver-node-gnrrz\" (UID: \"1375ec1e-4106-47ba-867a-42a96ed1014c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnrrz"
Sep 18 13:58:52.123635 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122308    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/58a315d7-c375-4d9d-9b8b-fe148b1329e1-cni-binary-copy\") pod \"multus-additional-cni-plugins-p5f5h\" (UID: \"58a315d7-c375-4d9d-9b8b-fe148b1329e1\") " pod="openshift-multus/multus-additional-cni-plugins-p5f5h"
Sep 18 13:58:52.123635 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122313    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f2201342-d012-4f49-aef5-4ca6901824d1-etc-modprobe-d\") pod \"tuned-bthf7\" (UID: \"f2201342-d012-4f49-aef5-4ca6901824d1\") " pod="openshift-cluster-node-tuning-operator/tuned-bthf7"
Sep 18 13:58:52.123635 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122331    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f2201342-d012-4f49-aef5-4ca6901824d1-etc-tuned\") pod \"tuned-bthf7\" (UID: \"f2201342-d012-4f49-aef5-4ca6901824d1\") " pod="openshift-cluster-node-tuning-operator/tuned-bthf7"
Sep 18 13:58:52.123635 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122357    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2c6e66f3-7060-4ece-823b-1a90f55768b3-host-run-multus-certs\") pod \"multus-czbmm\" (UID: \"2c6e66f3-7060-4ece-823b-1a90f55768b3\") " pod="openshift-multus/multus-czbmm"
Sep 18 13:58:52.123635 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122365    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c6e66f3-7060-4ece-823b-1a90f55768b3-cnibin\") pod \"multus-czbmm\" (UID: \"2c6e66f3-7060-4ece-823b-1a90f55768b3\") " pod="openshift-multus/multus-czbmm"
Sep 18 13:58:52.123635 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122368    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ca72836e-7ed0-47e5-a106-2f47340eb6b1-dbus\") pod \"global-pull-secret-syncer-2hvmm\" (UID: \"ca72836e-7ed0-47e5-a106-2f47340eb6b1\") " pod="kube-system/global-pull-secret-syncer-2hvmm"
Sep 18 13:58:52.123635 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122380    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ca72836e-7ed0-47e5-a106-2f47340eb6b1-original-pull-secret\") pod \"global-pull-secret-syncer-2hvmm\" (UID: \"ca72836e-7ed0-47e5-a106-2f47340eb6b1\") " pod="kube-system/global-pull-secret-syncer-2hvmm"
Sep 18 13:58:52.123635 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122406    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2201342-d012-4f49-aef5-4ca6901824d1-var-lib-kubelet\") pod \"tuned-bthf7\" (UID: \"f2201342-d012-4f49-aef5-4ca6901824d1\") " pod="openshift-cluster-node-tuning-operator/tuned-bthf7"
Sep 18 13:58:52.124480 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122431    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ca72836e-7ed0-47e5-a106-2f47340eb6b1-kubelet-config\") pod \"global-pull-secret-syncer-2hvmm\" (UID: \"ca72836e-7ed0-47e5-a106-2f47340eb6b1\") " pod="kube-system/global-pull-secret-syncer-2hvmm"
Sep 18 13:58:52.124480 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122457    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f2201342-d012-4f49-aef5-4ca6901824d1-etc-sysctl-conf\") pod \"tuned-bthf7\" (UID: \"f2201342-d012-4f49-aef5-4ca6901824d1\") " pod="openshift-cluster-node-tuning-operator/tuned-bthf7"
Sep 18 13:58:52.124480 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122481    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2c6e66f3-7060-4ece-823b-1a90f55768b3-host-run-netns\") pod \"multus-czbmm\" (UID: \"2c6e66f3-7060-4ece-823b-1a90f55768b3\") " pod="openshift-multus/multus-czbmm"
Sep 18 13:58:52.124480 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122503    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c6e66f3-7060-4ece-823b-1a90f55768b3-host-var-lib-kubelet\") pod \"multus-czbmm\" (UID: \"2c6e66f3-7060-4ece-823b-1a90f55768b3\") " pod="openshift-multus/multus-czbmm"
Sep 18 13:58:52.124480 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122513    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9f51a32f-f091-49b2-97f8-7e3ba0008474-konnectivity-ca\") pod \"konnectivity-agent-fkrgg\" (UID: \"9f51a32f-f091-49b2-97f8-7e3ba0008474\") " pod="kube-system/konnectivity-agent-fkrgg"
Sep 18 13:58:52.124480 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122525    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f2201342-d012-4f49-aef5-4ca6901824d1-etc-sysctl-d\") pod \"tuned-bthf7\" (UID: \"f2201342-d012-4f49-aef5-4ca6901824d1\") " pod="openshift-cluster-node-tuning-operator/tuned-bthf7"
Sep 18 13:58:52.124480 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122549    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f2201342-d012-4f49-aef5-4ca6901824d1-etc-systemd\") pod \"tuned-bthf7\" (UID: \"f2201342-d012-4f49-aef5-4ca6901824d1\") " pod="openshift-cluster-node-tuning-operator/tuned-bthf7"
Sep 18 13:58:52.124480 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122568    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2201342-d012-4f49-aef5-4ca6901824d1-etc-kubernetes\") pod \"tuned-bthf7\" (UID: \"f2201342-d012-4f49-aef5-4ca6901824d1\") " pod="openshift-cluster-node-tuning-operator/tuned-bthf7"
Sep 18 13:58:52.124480 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122572    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1375ec1e-4106-47ba-867a-42a96ed1014c-registration-dir\") pod \"aws-ebs-csi-driver-node-gnrrz\" (UID: \"1375ec1e-4106-47ba-867a-42a96ed1014c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnrrz"
Sep 18 13:58:52.124480 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122600    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58a315d7-c375-4d9d-9b8b-fe148b1329e1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p5f5h\" (UID: \"58a315d7-c375-4d9d-9b8b-fe148b1329e1\") " pod="openshift-multus/multus-additional-cni-plugins-p5f5h"
Sep 18 13:58:52.124480 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122621    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/58a315d7-c375-4d9d-9b8b-fe148b1329e1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p5f5h\" (UID: \"58a315d7-c375-4d9d-9b8b-fe148b1329e1\") " pod="openshift-multus/multus-additional-cni-plugins-p5f5h"
Sep 18 13:58:52.124480 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122642    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htmqx\" (UniqueName: \"kubernetes.io/projected/58a315d7-c375-4d9d-9b8b-fe148b1329e1-kube-api-access-htmqx\") pod \"multus-additional-cni-plugins-p5f5h\" (UID: \"58a315d7-c375-4d9d-9b8b-fe148b1329e1\") " pod="openshift-multus/multus-additional-cni-plugins-p5f5h"
Sep 18 13:58:52.124480 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122664    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2c6e66f3-7060-4ece-823b-1a90f55768b3-cni-binary-copy\") pod \"multus-czbmm\" (UID: \"2c6e66f3-7060-4ece-823b-1a90f55768b3\") " pod="openshift-multus/multus-czbmm"
Sep 18 13:58:52.124480 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122679    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f2201342-d012-4f49-aef5-4ca6901824d1-etc-sysctl-conf\") pod \"tuned-bthf7\" (UID: \"f2201342-d012-4f49-aef5-4ca6901824d1\") " pod="openshift-cluster-node-tuning-operator/tuned-bthf7"
Sep 18 13:58:52.124480 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122666    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c6e66f3-7060-4ece-823b-1a90f55768b3-multus-cni-dir\") pod \"multus-czbmm\" (UID: \"2c6e66f3-7060-4ece-823b-1a90f55768b3\") " pod="openshift-multus/multus-czbmm"
Sep 18 13:58:52.124480 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122718    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2c6e66f3-7060-4ece-823b-1a90f55768b3-host-run-multus-certs\") pod \"multus-czbmm\" (UID: \"2c6e66f3-7060-4ece-823b-1a90f55768b3\") " pod="openshift-multus/multus-czbmm"
Sep 18 13:58:52.124480 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122761    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ca72836e-7ed0-47e5-a106-2f47340eb6b1-kubelet-config\") pod \"global-pull-secret-syncer-2hvmm\" (UID: \"ca72836e-7ed0-47e5-a106-2f47340eb6b1\") " pod="kube-system/global-pull-secret-syncer-2hvmm"
Sep 18 13:58:52.125064 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122764    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f2201342-d012-4f49-aef5-4ca6901824d1-etc-sysconfig\") pod \"tuned-bthf7\" (UID: \"f2201342-d012-4f49-aef5-4ca6901824d1\") " pod="openshift-cluster-node-tuning-operator/tuned-bthf7"
Sep 18 13:58:52.125064 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122772    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2201342-d012-4f49-aef5-4ca6901824d1-var-lib-kubelet\") pod \"tuned-bthf7\" (UID: \"f2201342-d012-4f49-aef5-4ca6901824d1\") " pod="openshift-cluster-node-tuning-operator/tuned-bthf7"
Sep 18 13:58:52.125064 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122794    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/58a315d7-c375-4d9d-9b8b-fe148b1329e1-cni-binary-copy\") pod \"multus-additional-cni-plugins-p5f5h\" (UID: \"58a315d7-c375-4d9d-9b8b-fe148b1329e1\") " pod="openshift-multus/multus-additional-cni-plugins-p5f5h"
Sep 18 13:58:52.125064 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122811    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f2201342-d012-4f49-aef5-4ca6901824d1-etc-systemd\") pod \"tuned-bthf7\" (UID: \"f2201342-d012-4f49-aef5-4ca6901824d1\") " pod="openshift-cluster-node-tuning-operator/tuned-bthf7"
Sep 18 13:58:52.125064 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122826    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c6e66f3-7060-4ece-823b-1a90f55768b3-host-var-lib-kubelet\") pod \"multus-czbmm\" (UID: \"2c6e66f3-7060-4ece-823b-1a90f55768b3\") " pod="openshift-multus/multus-czbmm"
Sep 18 13:58:52.125064 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:52.122831    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 13:58:52.125064 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122842    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2c6e66f3-7060-4ece-823b-1a90f55768b3-host-run-netns\") pod \"multus-czbmm\" (UID: \"2c6e66f3-7060-4ece-823b-1a90f55768b3\") " pod="openshift-multus/multus-czbmm"
Sep 18 13:58:52.125064 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122886    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1375ec1e-4106-47ba-867a-42a96ed1014c-socket-dir\") pod \"aws-ebs-csi-driver-node-gnrrz\" (UID: \"1375ec1e-4106-47ba-867a-42a96ed1014c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnrrz"
Sep 18 13:58:52.125064 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122765    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f2201342-d012-4f49-aef5-4ca6901824d1-etc-sysctl-d\") pod \"tuned-bthf7\" (UID: \"f2201342-d012-4f49-aef5-4ca6901824d1\") " pod="openshift-cluster-node-tuning-operator/tuned-bthf7"
Sep 18 13:58:52.125064 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:52.122896    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca72836e-7ed0-47e5-a106-2f47340eb6b1-original-pull-secret podName:ca72836e-7ed0-47e5-a106-2f47340eb6b1 nodeName:}" failed. No retries permitted until 2026-09-18 13:58:52.622878484 +0000 UTC m=+4.511594395 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ca72836e-7ed0-47e5-a106-2f47340eb6b1-original-pull-secret") pod "global-pull-secret-syncer-2hvmm" (UID: "ca72836e-7ed0-47e5-a106-2f47340eb6b1") : object "kube-system"/"original-pull-secret" not registered
Sep 18 13:58:52.125064 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122950    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c6e66f3-7060-4ece-823b-1a90f55768b3-multus-cni-dir\") pod \"multus-czbmm\" (UID: \"2c6e66f3-7060-4ece-823b-1a90f55768b3\") " pod="openshift-multus/multus-czbmm"
Sep 18 13:58:52.125064 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.122887    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1375ec1e-4106-47ba-867a-42a96ed1014c-registration-dir\") pod \"aws-ebs-csi-driver-node-gnrrz\" (UID: \"1375ec1e-4106-47ba-867a-42a96ed1014c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnrrz"
Sep 18 13:58:52.125064 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.123003    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58a315d7-c375-4d9d-9b8b-fe148b1329e1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p5f5h\" (UID: \"58a315d7-c375-4d9d-9b8b-fe148b1329e1\") " pod="openshift-multus/multus-additional-cni-plugins-p5f5h"
Sep 18 13:58:52.125064 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.123211    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/58a315d7-c375-4d9d-9b8b-fe148b1329e1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p5f5h\" (UID: \"58a315d7-c375-4d9d-9b8b-fe148b1329e1\") " pod="openshift-multus/multus-additional-cni-plugins-p5f5h"
Sep 18 13:58:52.125064 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.124559    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9f51a32f-f091-49b2-97f8-7e3ba0008474-agent-certs\") pod \"konnectivity-agent-fkrgg\" (UID: \"9f51a32f-f091-49b2-97f8-7e3ba0008474\") " pod="kube-system/konnectivity-agent-fkrgg"
Sep 18 13:58:52.125064 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.124689    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2201342-d012-4f49-aef5-4ca6901824d1-tmp\") pod \"tuned-bthf7\" (UID: \"f2201342-d012-4f49-aef5-4ca6901824d1\") " pod="openshift-cluster-node-tuning-operator/tuned-bthf7"
Sep 18 13:58:52.125064 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.124916    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f2201342-d012-4f49-aef5-4ca6901824d1-etc-tuned\") pod \"tuned-bthf7\" (UID: \"f2201342-d012-4f49-aef5-4ca6901824d1\") " pod="openshift-cluster-node-tuning-operator/tuned-bthf7"
Sep 18 13:58:52.129886 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.129840    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcnx9\" (UniqueName: \"kubernetes.io/projected/2c6e66f3-7060-4ece-823b-1a90f55768b3-kube-api-access-tcnx9\") pod \"multus-czbmm\" (UID: \"2c6e66f3-7060-4ece-823b-1a90f55768b3\") " pod="openshift-multus/multus-czbmm"
Sep 18 13:58:52.130323 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.130263    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm9qk\" (UniqueName: \"kubernetes.io/projected/7f65deed-0197-489a-a602-062486b50a57-kube-api-access-xm9qk\") pod \"iptables-alerter-469c7\" (UID: \"7f65deed-0197-489a-a602-062486b50a57\") " pod="openshift-network-operator/iptables-alerter-469c7"
Sep 18 13:58:52.130323 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.130273    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl6x2\" (UniqueName: \"kubernetes.io/projected/f2201342-d012-4f49-aef5-4ca6901824d1-kube-api-access-vl6x2\") pod \"tuned-bthf7\" (UID: \"f2201342-d012-4f49-aef5-4ca6901824d1\") " pod="openshift-cluster-node-tuning-operator/tuned-bthf7"
Sep 18 13:58:52.130464 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.130388    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kqlh\" (UniqueName: \"kubernetes.io/projected/1375ec1e-4106-47ba-867a-42a96ed1014c-kube-api-access-5kqlh\") pod \"aws-ebs-csi-driver-node-gnrrz\" (UID: \"1375ec1e-4106-47ba-867a-42a96ed1014c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnrrz"
Sep 18 13:58:52.132547 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.132524    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htmqx\" (UniqueName: \"kubernetes.io/projected/58a315d7-c375-4d9d-9b8b-fe148b1329e1-kube-api-access-htmqx\") pod \"multus-additional-cni-plugins-p5f5h\" (UID: \"58a315d7-c375-4d9d-9b8b-fe148b1329e1\") " pod="openshift-multus/multus-additional-cni-plugins-p5f5h"
Sep 18 13:58:52.217324 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.217271    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t9ks5"
Sep 18 13:58:52.224747 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.224731    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8n79p"
Sep 18 13:58:52.237483 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.237465    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:58:52.242979 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.242965    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-czbmm"
Sep 18 13:58:52.252234 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.252218    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-469c7"
Sep 18 13:58:52.259886 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.259873    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bthf7"
Sep 18 13:58:52.267406 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.267390    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fkrgg"
Sep 18 13:58:52.273976 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.273961    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 13:58:52.275036 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.275020    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-p5f5h"
Sep 18 13:58:52.280665 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.280648    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnrrz"
Sep 18 13:58:52.525335 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.525247    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57282ff1-fab4-4cd4-8087-5517bd249f10-metrics-certs\") pod \"network-metrics-daemon-2mgfl\" (UID: \"57282ff1-fab4-4cd4-8087-5517bd249f10\") " pod="openshift-multus/network-metrics-daemon-2mgfl"
Sep 18 13:58:52.525494 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:52.525418    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:58:52.525494 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:52.525486    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57282ff1-fab4-4cd4-8087-5517bd249f10-metrics-certs podName:57282ff1-fab4-4cd4-8087-5517bd249f10 nodeName:}" failed. No retries permitted until 2026-09-18 13:58:53.525467849 +0000 UTC m=+5.414183760 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57282ff1-fab4-4cd4-8087-5517bd249f10-metrics-certs") pod "network-metrics-daemon-2mgfl" (UID: "57282ff1-fab4-4cd4-8087-5517bd249f10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:58:52.625548 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.625515    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t8sk\" (UniqueName: \"kubernetes.io/projected/7e8de495-7804-4c38-af57-655459217f1c-kube-api-access-4t8sk\") pod \"network-check-target-j7msq\" (UID: \"7e8de495-7804-4c38-af57-655459217f1c\") " pod="openshift-network-diagnostics/network-check-target-j7msq"
Sep 18 13:58:52.625708 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.625557    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ca72836e-7ed0-47e5-a106-2f47340eb6b1-original-pull-secret\") pod \"global-pull-secret-syncer-2hvmm\" (UID: \"ca72836e-7ed0-47e5-a106-2f47340eb6b1\") " pod="kube-system/global-pull-secret-syncer-2hvmm"
Sep 18 13:58:52.625708 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:52.625682    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 13:58:52.625708 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:52.625688    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 13:58:52.625708 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:52.625700    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 13:58:52.625708 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:52.625710    2579 projected.go:196] Error preparing data for projected volume kube-api-access-4t8sk for pod openshift-network-diagnostics/network-check-target-j7msq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:58:52.625949 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:52.625754    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca72836e-7ed0-47e5-a106-2f47340eb6b1-original-pull-secret podName:ca72836e-7ed0-47e5-a106-2f47340eb6b1 nodeName:}" failed. No retries permitted until 2026-09-18 13:58:53.625734334 +0000 UTC m=+5.514450256 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ca72836e-7ed0-47e5-a106-2f47340eb6b1-original-pull-secret") pod "global-pull-secret-syncer-2hvmm" (UID: "ca72836e-7ed0-47e5-a106-2f47340eb6b1") : object "kube-system"/"original-pull-secret" not registered
Sep 18 13:58:52.625949 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:52.625773    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7e8de495-7804-4c38-af57-655459217f1c-kube-api-access-4t8sk podName:7e8de495-7804-4c38-af57-655459217f1c nodeName:}" failed. No retries permitted until 2026-09-18 13:58:53.62576392 +0000 UTC m=+5.514479832 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4t8sk" (UniqueName: "kubernetes.io/projected/7e8de495-7804-4c38-af57-655459217f1c-kube-api-access-4t8sk") pod "network-check-target-j7msq" (UID: "7e8de495-7804-4c38-af57-655459217f1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:58:52.909882 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:52.909855    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod690b430b_1984_4119_beae_85786c10b267.slice/crio-6427a07a7873e1f10c12446f494e8a44da1737cae827e6f0d8c944d2357a9b7b WatchSource:0}: Error finding container 6427a07a7873e1f10c12446f494e8a44da1737cae827e6f0d8c944d2357a9b7b: Status 404 returned error can't find the container with id 6427a07a7873e1f10c12446f494e8a44da1737cae827e6f0d8c944d2357a9b7b
Sep 18 13:58:52.914289 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:52.914109    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4626a667_4cd6_468c_8ef3_6e3a925c2608.slice/crio-7d3c65f8fdf5a81d3a3b27a3cd4c8ee5561fa81d16cc919d0c3b59093ef87259 WatchSource:0}: Error finding container 7d3c65f8fdf5a81d3a3b27a3cd4c8ee5561fa81d16cc919d0c3b59093ef87259: Status 404 returned error can't find the container with id 7d3c65f8fdf5a81d3a3b27a3cd4c8ee5561fa81d16cc919d0c3b59093ef87259
Sep 18 13:58:52.915791 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:52.915768    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbb463f3_8dcd_413d_859f_5dcf87c59115.slice/crio-f03c507e96be5e7b19f6687737946b80f47fbda99d5901e034bad814ceca9720 WatchSource:0}: Error finding container f03c507e96be5e7b19f6687737946b80f47fbda99d5901e034bad814ceca9720: Status 404 returned error can't find the container with id f03c507e96be5e7b19f6687737946b80f47fbda99d5901e034bad814ceca9720
Sep 18 13:58:52.917530 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:52.917507    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f65deed_0197_489a_a602_062486b50a57.slice/crio-8b55a3642d9062561b09f9957ed74929d96eae0f01c22419fa7fb81aa6f729c8 WatchSource:0}: Error finding container 8b55a3642d9062561b09f9957ed74929d96eae0f01c22419fa7fb81aa6f729c8: Status 404 returned error can't find the container with id 8b55a3642d9062561b09f9957ed74929d96eae0f01c22419fa7fb81aa6f729c8
Sep 18 13:58:52.919121 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:52.919031    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a315d7_c375_4d9d_9b8b_fe148b1329e1.slice/crio-dd774b3f09d2b65fd8e5b8aed66872d64d78c0fdab607495056a241d65871d09 WatchSource:0}: Error finding container dd774b3f09d2b65fd8e5b8aed66872d64d78c0fdab607495056a241d65871d09: Status 404 returned error can't find the container with id dd774b3f09d2b65fd8e5b8aed66872d64d78c0fdab607495056a241d65871d09
Sep 18 13:58:52.920046 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:52.920016    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2201342_d012_4f49_aef5_4ca6901824d1.slice/crio-cfbdf5de2a4b77d543d98175b1d8ac550959d667d4e40658f1327fc2466ddf0e WatchSource:0}: Error finding container cfbdf5de2a4b77d543d98175b1d8ac550959d667d4e40658f1327fc2466ddf0e: Status 404 returned error can't find the container with id cfbdf5de2a4b77d543d98175b1d8ac550959d667d4e40658f1327fc2466ddf0e
Sep 18 13:58:52.921768 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:52.921701    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f51a32f_f091_49b2_97f8_7e3ba0008474.slice/crio-90c58d974477d5679695bf1cc2d0c6d882c56c5b7c67be37e3efa38114082b7a WatchSource:0}: Error finding container 90c58d974477d5679695bf1cc2d0c6d882c56c5b7c67be37e3efa38114082b7a: Status 404 returned error can't find the container with id 90c58d974477d5679695bf1cc2d0c6d882c56c5b7c67be37e3efa38114082b7a
Sep 18 13:58:52.923038 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:52.922845    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1375ec1e_4106_47ba_867a_42a96ed1014c.slice/crio-b0124d52e549b22d4a8f2e4df404f305c1df336a375a3a47a278a65144d5cb48 WatchSource:0}: Error finding container b0124d52e549b22d4a8f2e4df404f305c1df336a375a3a47a278a65144d5cb48: Status 404 returned error can't find the container with id b0124d52e549b22d4a8f2e4df404f305c1df336a375a3a47a278a65144d5cb48
Sep 18 13:58:52.923625 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:58:52.923582    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c6e66f3_7060_4ece_823b_1a90f55768b3.slice/crio-49ae78db420b636a5c79b6d2091800ce7a86b7c59f3b9013b8907ed1a5a1c5bc WatchSource:0}: Error finding container 49ae78db420b636a5c79b6d2091800ce7a86b7c59f3b9013b8907ed1a5a1c5bc: Status 404 returned error can't find the container with id 49ae78db420b636a5c79b6d2091800ce7a86b7c59f3b9013b8907ed1a5a1c5bc
Sep 18 13:58:52.960142 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.960113    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 13:53:51 +0000 UTC" deadline="2028-05-15 15:54:11.945935638 +0000 UTC"
Sep 18 13:58:52.960142 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:52.960141    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14521h55m18.985798404s"
Sep 18 13:58:53.029521 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:53.029494    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7msq"
Sep 18 13:58:53.029614 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:53.029534    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2hvmm"
Sep 18 13:58:53.029663 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:53.029623    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j7msq" podUID="7e8de495-7804-4c38-af57-655459217f1c"
Sep 18 13:58:53.029775 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:53.029755    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2hvmm" podUID="ca72836e-7ed0-47e5-a106-2f47340eb6b1"
Sep 18 13:58:53.037753 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:53.037728    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-469c7" event={"ID":"7f65deed-0197-489a-a602-062486b50a57","Type":"ContainerStarted","Data":"8b55a3642d9062561b09f9957ed74929d96eae0f01c22419fa7fb81aa6f729c8"}
Sep 18 13:58:53.043413 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:53.043390    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-120.ec2.internal" event={"ID":"59194d24519fb418bb58aa767b6c58b0","Type":"ContainerStarted","Data":"2fffc38ee094dab1e1115b808b9f40dcfd88b2be16033e48d92d90097103992d"}
Sep 18 13:58:53.045327 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:53.045308    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bthf7" event={"ID":"f2201342-d012-4f49-aef5-4ca6901824d1","Type":"ContainerStarted","Data":"cfbdf5de2a4b77d543d98175b1d8ac550959d667d4e40658f1327fc2466ddf0e"}
Sep 18 13:58:53.047959 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:53.047938    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p5f5h" event={"ID":"58a315d7-c375-4d9d-9b8b-fe148b1329e1","Type":"ContainerStarted","Data":"dd774b3f09d2b65fd8e5b8aed66872d64d78c0fdab607495056a241d65871d09"}
Sep 18 13:58:53.049697 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:53.049676    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8n79p" event={"ID":"dbb463f3-8dcd-413d-859f-5dcf87c59115","Type":"ContainerStarted","Data":"f03c507e96be5e7b19f6687737946b80f47fbda99d5901e034bad814ceca9720"}
Sep 18 13:58:53.050752 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:53.050730    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv" event={"ID":"4626a667-4cd6-468c-8ef3-6e3a925c2608","Type":"ContainerStarted","Data":"7d3c65f8fdf5a81d3a3b27a3cd4c8ee5561fa81d16cc919d0c3b59093ef87259"}
Sep 18 13:58:53.051734 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:53.051715    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t9ks5" event={"ID":"690b430b-1984-4119-beae-85786c10b267","Type":"ContainerStarted","Data":"6427a07a7873e1f10c12446f494e8a44da1737cae827e6f0d8c944d2357a9b7b"}
Sep 18 13:58:53.053221 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:53.053198    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-czbmm" event={"ID":"2c6e66f3-7060-4ece-823b-1a90f55768b3","Type":"ContainerStarted","Data":"49ae78db420b636a5c79b6d2091800ce7a86b7c59f3b9013b8907ed1a5a1c5bc"}
Sep 18 13:58:53.054214 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:53.054194    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnrrz" event={"ID":"1375ec1e-4106-47ba-867a-42a96ed1014c","Type":"ContainerStarted","Data":"b0124d52e549b22d4a8f2e4df404f305c1df336a375a3a47a278a65144d5cb48"}
Sep 18 13:58:53.055124 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:53.055107    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fkrgg" event={"ID":"9f51a32f-f091-49b2-97f8-7e3ba0008474","Type":"ContainerStarted","Data":"90c58d974477d5679695bf1cc2d0c6d882c56c5b7c67be37e3efa38114082b7a"}
Sep 18 13:58:53.532542 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:53.532458    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57282ff1-fab4-4cd4-8087-5517bd249f10-metrics-certs\") pod \"network-metrics-daemon-2mgfl\" (UID: \"57282ff1-fab4-4cd4-8087-5517bd249f10\") " pod="openshift-multus/network-metrics-daemon-2mgfl"
Sep 18 13:58:53.532693 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:53.532643    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:58:53.532753 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:53.532700    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57282ff1-fab4-4cd4-8087-5517bd249f10-metrics-certs podName:57282ff1-fab4-4cd4-8087-5517bd249f10 nodeName:}" failed. No retries permitted until 2026-09-18 13:58:55.532682304 +0000 UTC m=+7.421398229 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57282ff1-fab4-4cd4-8087-5517bd249f10-metrics-certs") pod "network-metrics-daemon-2mgfl" (UID: "57282ff1-fab4-4cd4-8087-5517bd249f10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:58:53.633630 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:53.633597    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t8sk\" (UniqueName: \"kubernetes.io/projected/7e8de495-7804-4c38-af57-655459217f1c-kube-api-access-4t8sk\") pod \"network-check-target-j7msq\" (UID: \"7e8de495-7804-4c38-af57-655459217f1c\") " pod="openshift-network-diagnostics/network-check-target-j7msq"
Sep 18 13:58:53.633793 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:53.633645    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ca72836e-7ed0-47e5-a106-2f47340eb6b1-original-pull-secret\") pod \"global-pull-secret-syncer-2hvmm\" (UID: \"ca72836e-7ed0-47e5-a106-2f47340eb6b1\") " pod="kube-system/global-pull-secret-syncer-2hvmm"
Sep 18 13:58:53.633860 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:53.633798    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 13:58:53.633860 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:53.633853    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca72836e-7ed0-47e5-a106-2f47340eb6b1-original-pull-secret podName:ca72836e-7ed0-47e5-a106-2f47340eb6b1 nodeName:}" failed. No retries permitted until 2026-09-18 13:58:55.633835385 +0000 UTC m=+7.522551297 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ca72836e-7ed0-47e5-a106-2f47340eb6b1-original-pull-secret") pod "global-pull-secret-syncer-2hvmm" (UID: "ca72836e-7ed0-47e5-a106-2f47340eb6b1") : object "kube-system"/"original-pull-secret" not registered
Sep 18 13:58:53.634263 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:53.634243    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 13:58:53.634263 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:53.634266    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 13:58:53.634421 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:53.634280    2579 projected.go:196] Error preparing data for projected volume kube-api-access-4t8sk for pod openshift-network-diagnostics/network-check-target-j7msq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:58:53.634421 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:53.634321    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7e8de495-7804-4c38-af57-655459217f1c-kube-api-access-4t8sk podName:7e8de495-7804-4c38-af57-655459217f1c nodeName:}" failed. No retries permitted until 2026-09-18 13:58:55.63430719 +0000 UTC m=+7.523023104 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4t8sk" (UniqueName: "kubernetes.io/projected/7e8de495-7804-4c38-af57-655459217f1c-kube-api-access-4t8sk") pod "network-check-target-j7msq" (UID: "7e8de495-7804-4c38-af57-655459217f1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:58:53.960804 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:53.960678    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 13:53:51 +0000 UTC" deadline="2028-06-26 18:35:50.962098486 +0000 UTC"
Sep 18 13:58:53.960804 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:53.960732    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15532h36m57.00136959s"
Sep 18 13:58:54.031642 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:54.031607    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2mgfl"
Sep 18 13:58:54.031786 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:54.031761    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2mgfl" podUID="57282ff1-fab4-4cd4-8087-5517bd249f10"
Sep 18 13:58:54.066000 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:54.065481    2579 generic.go:356] "Generic (PLEG): container finished" podID="596255870e755bf312825ba49bcccfdb" containerID="750ac5732c4307445c20a5d09bce9e1874ed6777e4d5d470a0de28c80d48b9ad" exitCode=0
Sep 18 13:58:54.066000 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:54.065578    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal" event={"ID":"596255870e755bf312825ba49bcccfdb","Type":"ContainerDied","Data":"750ac5732c4307445c20a5d09bce9e1874ed6777e4d5d470a0de28c80d48b9ad"}
Sep 18 13:58:54.106902 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:54.106848    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-120.ec2.internal" podStartSLOduration=3.106828962 podStartE2EDuration="3.106828962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:58:53.067313406 +0000 UTC m=+4.956029334" watchObservedRunningTime="2026-09-18 13:58:54.106828962 +0000 UTC m=+5.995544893"
Sep 18 13:58:55.029624 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:55.029590    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7msq"
Sep 18 13:58:55.030104 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:55.029711    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j7msq" podUID="7e8de495-7804-4c38-af57-655459217f1c"
Sep 18 13:58:55.030104 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:55.030077    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2hvmm"
Sep 18 13:58:55.030318 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:55.030190    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2hvmm" podUID="ca72836e-7ed0-47e5-a106-2f47340eb6b1"
Sep 18 13:58:55.093013 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:55.092984    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal" event={"ID":"596255870e755bf312825ba49bcccfdb","Type":"ContainerStarted","Data":"9194d3fd2380e969109b41b6c9e45e09f839662d157e68992557698c12bfdb29"}
Sep 18 13:58:55.549766 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:55.549732    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57282ff1-fab4-4cd4-8087-5517bd249f10-metrics-certs\") pod \"network-metrics-daemon-2mgfl\" (UID: \"57282ff1-fab4-4cd4-8087-5517bd249f10\") " pod="openshift-multus/network-metrics-daemon-2mgfl"
Sep 18 13:58:55.549955 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:55.549856    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:58:55.549955 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:55.549907    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57282ff1-fab4-4cd4-8087-5517bd249f10-metrics-certs podName:57282ff1-fab4-4cd4-8087-5517bd249f10 nodeName:}" failed. No retries permitted until 2026-09-18 13:58:59.549893686 +0000 UTC m=+11.438609594 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57282ff1-fab4-4cd4-8087-5517bd249f10-metrics-certs") pod "network-metrics-daemon-2mgfl" (UID: "57282ff1-fab4-4cd4-8087-5517bd249f10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:58:55.650396 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:55.650344    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t8sk\" (UniqueName: \"kubernetes.io/projected/7e8de495-7804-4c38-af57-655459217f1c-kube-api-access-4t8sk\") pod \"network-check-target-j7msq\" (UID: \"7e8de495-7804-4c38-af57-655459217f1c\") " pod="openshift-network-diagnostics/network-check-target-j7msq"
Sep 18 13:58:55.650396 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:55.650401    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ca72836e-7ed0-47e5-a106-2f47340eb6b1-original-pull-secret\") pod \"global-pull-secret-syncer-2hvmm\" (UID: \"ca72836e-7ed0-47e5-a106-2f47340eb6b1\") " pod="kube-system/global-pull-secret-syncer-2hvmm"
Sep 18 13:58:55.650666 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:55.650549    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 13:58:55.650666 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:55.650612    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca72836e-7ed0-47e5-a106-2f47340eb6b1-original-pull-secret podName:ca72836e-7ed0-47e5-a106-2f47340eb6b1 nodeName:}" failed. No retries permitted until 2026-09-18 13:58:59.650593101 +0000 UTC m=+11.539309012 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ca72836e-7ed0-47e5-a106-2f47340eb6b1-original-pull-secret") pod "global-pull-secret-syncer-2hvmm" (UID: "ca72836e-7ed0-47e5-a106-2f47340eb6b1") : object "kube-system"/"original-pull-secret" not registered
Sep 18 13:58:55.651094 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:55.651072    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 13:58:55.651094 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:55.651093    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 13:58:55.651287 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:55.651106    2579 projected.go:196] Error preparing data for projected volume kube-api-access-4t8sk for pod openshift-network-diagnostics/network-check-target-j7msq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:58:55.651287 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:55.651154    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7e8de495-7804-4c38-af57-655459217f1c-kube-api-access-4t8sk podName:7e8de495-7804-4c38-af57-655459217f1c nodeName:}" failed. No retries permitted until 2026-09-18 13:58:59.651138664 +0000 UTC m=+11.539854576 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4t8sk" (UniqueName: "kubernetes.io/projected/7e8de495-7804-4c38-af57-655459217f1c-kube-api-access-4t8sk") pod "network-check-target-j7msq" (UID: "7e8de495-7804-4c38-af57-655459217f1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:58:56.030089 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:56.030002    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2mgfl"
Sep 18 13:58:56.030554 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:56.030158    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2mgfl" podUID="57282ff1-fab4-4cd4-8087-5517bd249f10"
Sep 18 13:58:57.029413 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:57.029360    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7msq"
Sep 18 13:58:57.029413 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:57.029383    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2hvmm"
Sep 18 13:58:57.029651 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:57.029470    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j7msq" podUID="7e8de495-7804-4c38-af57-655459217f1c"
Sep 18 13:58:57.030009 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:57.029858    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2hvmm" podUID="ca72836e-7ed0-47e5-a106-2f47340eb6b1"
Sep 18 13:58:58.029647 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:58.029610    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2mgfl"
Sep 18 13:58:58.030054 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:58.029754    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2mgfl" podUID="57282ff1-fab4-4cd4-8087-5517bd249f10"
Sep 18 13:58:59.029643 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:59.029603    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7msq"
Sep 18 13:58:59.029798 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:59.029733    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j7msq" podUID="7e8de495-7804-4c38-af57-655459217f1c"
Sep 18 13:58:59.030096 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:59.029795    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2hvmm"
Sep 18 13:58:59.030096 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:59.029921    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2hvmm" podUID="ca72836e-7ed0-47e5-a106-2f47340eb6b1"
Sep 18 13:58:59.582934 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:59.582899    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57282ff1-fab4-4cd4-8087-5517bd249f10-metrics-certs\") pod \"network-metrics-daemon-2mgfl\" (UID: \"57282ff1-fab4-4cd4-8087-5517bd249f10\") " pod="openshift-multus/network-metrics-daemon-2mgfl"
Sep 18 13:58:59.583107 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:59.583053    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:58:59.583198 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:59.583122    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57282ff1-fab4-4cd4-8087-5517bd249f10-metrics-certs podName:57282ff1-fab4-4cd4-8087-5517bd249f10 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:07.583101977 +0000 UTC m=+19.471817901 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57282ff1-fab4-4cd4-8087-5517bd249f10-metrics-certs") pod "network-metrics-daemon-2mgfl" (UID: "57282ff1-fab4-4cd4-8087-5517bd249f10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:58:59.684117 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:59.684086    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t8sk\" (UniqueName: \"kubernetes.io/projected/7e8de495-7804-4c38-af57-655459217f1c-kube-api-access-4t8sk\") pod \"network-check-target-j7msq\" (UID: \"7e8de495-7804-4c38-af57-655459217f1c\") " pod="openshift-network-diagnostics/network-check-target-j7msq"
Sep 18 13:58:59.684322 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:58:59.684126    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ca72836e-7ed0-47e5-a106-2f47340eb6b1-original-pull-secret\") pod \"global-pull-secret-syncer-2hvmm\" (UID: \"ca72836e-7ed0-47e5-a106-2f47340eb6b1\") " pod="kube-system/global-pull-secret-syncer-2hvmm"
Sep 18 13:58:59.684322 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:59.684269    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 13:58:59.684475 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:59.684339    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca72836e-7ed0-47e5-a106-2f47340eb6b1-original-pull-secret podName:ca72836e-7ed0-47e5-a106-2f47340eb6b1 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:07.68432064 +0000 UTC m=+19.573036563 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ca72836e-7ed0-47e5-a106-2f47340eb6b1-original-pull-secret") pod "global-pull-secret-syncer-2hvmm" (UID: "ca72836e-7ed0-47e5-a106-2f47340eb6b1") : object "kube-system"/"original-pull-secret" not registered
Sep 18 13:58:59.684653 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:59.684273    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 13:58:59.684721 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:59.684670    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 13:58:59.684721 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:59.684687    2579 projected.go:196] Error preparing data for projected volume kube-api-access-4t8sk for pod openshift-network-diagnostics/network-check-target-j7msq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:58:59.684820 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:58:59.684732    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7e8de495-7804-4c38-af57-655459217f1c-kube-api-access-4t8sk podName:7e8de495-7804-4c38-af57-655459217f1c nodeName:}" failed. No retries permitted until 2026-09-18 13:59:07.684717906 +0000 UTC m=+19.573433828 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4t8sk" (UniqueName: "kubernetes.io/projected/7e8de495-7804-4c38-af57-655459217f1c-kube-api-access-4t8sk") pod "network-check-target-j7msq" (UID: "7e8de495-7804-4c38-af57-655459217f1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:59:00.030904 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:00.030299    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2mgfl"
Sep 18 13:59:00.030904 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:00.030425    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2mgfl" podUID="57282ff1-fab4-4cd4-8087-5517bd249f10"
Sep 18 13:59:01.029681 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:01.029446    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7msq"
Sep 18 13:59:01.029851 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:01.029783    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j7msq" podUID="7e8de495-7804-4c38-af57-655459217f1c"
Sep 18 13:59:01.029913 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:01.029576    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2hvmm"
Sep 18 13:59:01.029971 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:01.029946    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2hvmm" podUID="ca72836e-7ed0-47e5-a106-2f47340eb6b1"
Sep 18 13:59:02.029984 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:02.029943    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2mgfl"
Sep 18 13:59:02.030454 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:02.030088    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2mgfl" podUID="57282ff1-fab4-4cd4-8087-5517bd249f10"
Sep 18 13:59:03.029131 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:03.029046    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2hvmm"
Sep 18 13:59:03.029131 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:03.029083    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7msq"
Sep 18 13:59:03.029360 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:03.029185    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2hvmm" podUID="ca72836e-7ed0-47e5-a106-2f47340eb6b1"
Sep 18 13:59:03.029360 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:03.029291    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j7msq" podUID="7e8de495-7804-4c38-af57-655459217f1c"
Sep 18 13:59:04.030016 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:04.029984    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2mgfl"
Sep 18 13:59:04.030557 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:04.030117    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2mgfl" podUID="57282ff1-fab4-4cd4-8087-5517bd249f10"
Sep 18 13:59:05.029627 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:05.029601    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2hvmm"
Sep 18 13:59:05.029781 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:05.029640    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7msq"
Sep 18 13:59:05.029781 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:05.029692    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2hvmm" podUID="ca72836e-7ed0-47e5-a106-2f47340eb6b1"
Sep 18 13:59:05.029892 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:05.029819    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j7msq" podUID="7e8de495-7804-4c38-af57-655459217f1c"
Sep 18 13:59:06.029120 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:06.029090    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2mgfl"
Sep 18 13:59:06.029522 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:06.029256    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2mgfl" podUID="57282ff1-fab4-4cd4-8087-5517bd249f10"
Sep 18 13:59:07.029116 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:07.029087    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2hvmm"
Sep 18 13:59:07.029342 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:07.029222    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2hvmm" podUID="ca72836e-7ed0-47e5-a106-2f47340eb6b1"
Sep 18 13:59:07.029342 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:07.029279    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7msq"
Sep 18 13:59:07.029710 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:07.029389    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j7msq" podUID="7e8de495-7804-4c38-af57-655459217f1c"
Sep 18 13:59:07.645355 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:07.645326    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57282ff1-fab4-4cd4-8087-5517bd249f10-metrics-certs\") pod \"network-metrics-daemon-2mgfl\" (UID: \"57282ff1-fab4-4cd4-8087-5517bd249f10\") " pod="openshift-multus/network-metrics-daemon-2mgfl"
Sep 18 13:59:07.645526 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:07.645461    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:59:07.645526 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:07.645510    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57282ff1-fab4-4cd4-8087-5517bd249f10-metrics-certs podName:57282ff1-fab4-4cd4-8087-5517bd249f10 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:23.645497289 +0000 UTC m=+35.534213203 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57282ff1-fab4-4cd4-8087-5517bd249f10-metrics-certs") pod "network-metrics-daemon-2mgfl" (UID: "57282ff1-fab4-4cd4-8087-5517bd249f10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:59:07.745768 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:07.745736    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t8sk\" (UniqueName: \"kubernetes.io/projected/7e8de495-7804-4c38-af57-655459217f1c-kube-api-access-4t8sk\") pod \"network-check-target-j7msq\" (UID: \"7e8de495-7804-4c38-af57-655459217f1c\") " pod="openshift-network-diagnostics/network-check-target-j7msq"
Sep 18 13:59:07.745909 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:07.745778    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ca72836e-7ed0-47e5-a106-2f47340eb6b1-original-pull-secret\") pod \"global-pull-secret-syncer-2hvmm\" (UID: \"ca72836e-7ed0-47e5-a106-2f47340eb6b1\") " pod="kube-system/global-pull-secret-syncer-2hvmm"
Sep 18 13:59:07.745953 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:07.745911    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 13:59:07.745953 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:07.745932    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 13:59:07.746054 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:07.745977    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca72836e-7ed0-47e5-a106-2f47340eb6b1-original-pull-secret podName:ca72836e-7ed0-47e5-a106-2f47340eb6b1 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:23.745964485 +0000 UTC m=+35.634680394 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ca72836e-7ed0-47e5-a106-2f47340eb6b1-original-pull-secret") pod "global-pull-secret-syncer-2hvmm" (UID: "ca72836e-7ed0-47e5-a106-2f47340eb6b1") : object "kube-system"/"original-pull-secret" not registered
Sep 18 13:59:07.746054 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:07.745934    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 13:59:07.746054 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:07.746038    2579 projected.go:196] Error preparing data for projected volume kube-api-access-4t8sk for pod openshift-network-diagnostics/network-check-target-j7msq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:59:07.746189 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:07.746111    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7e8de495-7804-4c38-af57-655459217f1c-kube-api-access-4t8sk podName:7e8de495-7804-4c38-af57-655459217f1c nodeName:}" failed. No retries permitted until 2026-09-18 13:59:23.746092158 +0000 UTC m=+35.634808078 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4t8sk" (UniqueName: "kubernetes.io/projected/7e8de495-7804-4c38-af57-655459217f1c-kube-api-access-4t8sk") pod "network-check-target-j7msq" (UID: "7e8de495-7804-4c38-af57-655459217f1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:59:08.029468 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:08.029392    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2mgfl"
Sep 18 13:59:08.029913 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:08.029525    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2mgfl" podUID="57282ff1-fab4-4cd4-8087-5517bd249f10"
Sep 18 13:59:09.029419 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:09.029387    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2hvmm"
Sep 18 13:59:09.029613 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:09.029387    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7msq"
Sep 18 13:59:09.029613 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:09.029496    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2hvmm" podUID="ca72836e-7ed0-47e5-a106-2f47340eb6b1"
Sep 18 13:59:09.030007 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:09.029605    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j7msq" podUID="7e8de495-7804-4c38-af57-655459217f1c"
Sep 18 13:59:10.030593 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:10.030569    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2mgfl"
Sep 18 13:59:10.030918 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:10.030645    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2mgfl" podUID="57282ff1-fab4-4cd4-8087-5517bd249f10"
Sep 18 13:59:11.029729 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:11.029507    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7msq"
Sep 18 13:59:11.029852 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:11.029815    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j7msq" podUID="7e8de495-7804-4c38-af57-655459217f1c"
Sep 18 13:59:11.030014 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:11.029559    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2hvmm"
Sep 18 13:59:11.030153 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:11.030130    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2hvmm" podUID="ca72836e-7ed0-47e5-a106-2f47340eb6b1"
Sep 18 13:59:11.126269 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:11.126202    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-czbmm" event={"ID":"2c6e66f3-7060-4ece-823b-1a90f55768b3","Type":"ContainerStarted","Data":"38ece52258be78008c58f2a8770416b5f79b9fcb2c5ba470147060b368132c80"}
Sep 18 13:59:11.127699 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:11.127676    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnrrz" event={"ID":"1375ec1e-4106-47ba-867a-42a96ed1014c","Type":"ContainerStarted","Data":"17a7a2f772f261b8df40fdc3b96e8cea5c2bfdbea7a2f3fac09d2f950c22b191"}
Sep 18 13:59:11.129052 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:11.129032    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fkrgg" event={"ID":"9f51a32f-f091-49b2-97f8-7e3ba0008474","Type":"ContainerStarted","Data":"7cba8750369f070db696bd8477e17ccefdcd6c35507bf293e8e75efb5784cf20"}
Sep 18 13:59:11.130538 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:11.130512    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bthf7" event={"ID":"f2201342-d012-4f49-aef5-4ca6901824d1","Type":"ContainerStarted","Data":"6b60f066b02a2ea4a4c18d0ed6e87d8ee0d840d853bc145a934e75b5bbd85d52"}
Sep 18 13:59:11.132366 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:11.132340    2579 generic.go:356] "Generic (PLEG): container finished" podID="58a315d7-c375-4d9d-9b8b-fe148b1329e1" containerID="55971fbf6e1aa18294c782b5af7fb1790f59ca04f7d4f3e2e2fa1dbd0b4546e8" exitCode=0
Sep 18 13:59:11.132462 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:11.132428    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p5f5h" event={"ID":"58a315d7-c375-4d9d-9b8b-fe148b1329e1","Type":"ContainerDied","Data":"55971fbf6e1aa18294c782b5af7fb1790f59ca04f7d4f3e2e2fa1dbd0b4546e8"}
Sep 18 13:59:11.134529 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:11.134512    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8n79p" event={"ID":"dbb463f3-8dcd-413d-859f-5dcf87c59115","Type":"ContainerStarted","Data":"1ad8f6ba6e3b08f8d26fce4908f5a75889ee2994aa7d17e6fce8c4f2651acc1d"}
Sep 18 13:59:11.137306 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:11.137291    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sg4tv_4626a667-4cd6-468c-8ef3-6e3a925c2608/ovn-acl-logging/0.log"
Sep 18 13:59:11.137658 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:11.137640    2579 generic.go:356] "Generic (PLEG): container finished" podID="4626a667-4cd6-468c-8ef3-6e3a925c2608" containerID="37a1a203681d866b3abbb049cbb00812a2da513d42777889300fc56b6640a2b5" exitCode=1
Sep 18 13:59:11.137727 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:11.137706    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv" event={"ID":"4626a667-4cd6-468c-8ef3-6e3a925c2608","Type":"ContainerStarted","Data":"51dbf2db2efbc18895be8b0c413825f76cafc897f40e1991aeee9ba127830b33"}
Sep 18 13:59:11.137789 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:11.137776    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv" event={"ID":"4626a667-4cd6-468c-8ef3-6e3a925c2608","Type":"ContainerStarted","Data":"b5927faf1241c7b998feec49636d4395a10c4cfd10851eeb940fb6ead8e74266"}
Sep 18 13:59:11.137840 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:11.137827    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv" event={"ID":"4626a667-4cd6-468c-8ef3-6e3a925c2608","Type":"ContainerStarted","Data":"290738f29cc1f2bfc25d7a850207f3b873d4eecdeae458dc4aa3a70e91bae0dd"}
Sep 18 13:59:11.137895 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:11.137877    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv" event={"ID":"4626a667-4cd6-468c-8ef3-6e3a925c2608","Type":"ContainerStarted","Data":"6390f150cdd48ca99a817229b043772b48fca377f63d05ffa994c86c44f19957"}
Sep 18 13:59:11.137936 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:11.137926    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv" event={"ID":"4626a667-4cd6-468c-8ef3-6e3a925c2608","Type":"ContainerDied","Data":"37a1a203681d866b3abbb049cbb00812a2da513d42777889300fc56b6640a2b5"}
Sep 18 13:59:11.137982 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:11.137968    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv" event={"ID":"4626a667-4cd6-468c-8ef3-6e3a925c2608","Type":"ContainerStarted","Data":"103c22ed88e840a8afb940ac507172f4ffd1996b81256ec778e789d6a6596218"}
Sep 18 13:59:11.139123 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:11.139102    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t9ks5" event={"ID":"690b430b-1984-4119-beae-85786c10b267","Type":"ContainerStarted","Data":"6c5111e1474357b79a2cc25450b7bfecd292255d7b80c2dab18daac5906bd4c1"}
Sep 18 13:59:11.162143 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:11.162100    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-120.ec2.internal" podStartSLOduration=20.162087894 podStartE2EDuration="20.162087894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:58:55.120589548 +0000 UTC m=+7.009305481" watchObservedRunningTime="2026-09-18 13:59:11.162087894 +0000 UTC m=+23.050803826"
Sep 18 13:59:11.197373 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:11.197337    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-t9ks5" podStartSLOduration=4.329363508 podStartE2EDuration="21.197325326s" totalImagesPullingTime="16.867961818s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:58:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:59:11.19639263 +0000 UTC m=+23.085108564" watchObservedRunningTime="2026-09-18 13:59:11.197325326 +0000 UTC m=+23.086041256"
Sep 18 13:59:11.197459 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:11.197446    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-czbmm" podStartSLOduration=3.881327306 podStartE2EDuration="21.197441136s" totalImagesPullingTime="17.31611383s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:58:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:59:11.162854202 +0000 UTC m=+23.051570133" watchObservedRunningTime="2026-09-18 13:59:11.197441136 +0000 UTC m=+23.086157063"
Sep 18 13:59:11.222346 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:11.222304    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8n79p" podStartSLOduration=3.943969822 podStartE2EDuration="21.222289506s" totalImagesPullingTime="17.278319684s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:58:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:59:11.219879056 +0000 UTC m=+23.108594987" watchObservedRunningTime="2026-09-18 13:59:11.222289506 +0000 UTC m=+23.111005438"
Sep 18 13:59:11.271413 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:11.271378    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-fkrgg" podStartSLOduration=8.255851527 podStartE2EDuration="21.271367148s" totalImagesPullingTime="13.015515621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:58:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:59:11.271108581 +0000 UTC m=+23.159824515" watchObservedRunningTime="2026-09-18 13:59:11.271367148 +0000 UTC m=+23.160083079"
Sep 18 13:59:11.271595 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:11.271563    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bthf7" podStartSLOduration=3.996215645 podStartE2EDuration="21.271555135s" totalImagesPullingTime="17.27533949s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:58:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:59:11.248763863 +0000 UTC m=+23.137479795" watchObservedRunningTime="2026-09-18 13:59:11.271555135 +0000 UTC m=+23.160271065"
Sep 18 13:59:11.408664 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:11.408608    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:11.506442 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:11.506424    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:11.713621 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:11.713480    2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 18 13:59:12.013325 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:12.013184    2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T13:59:11.713618699Z","UUID":"9b9af5b6-78e3-4ac0-a4b9-8feed33daf7a","Handler":null,"Name":"","Endpoint":""}
Sep 18 13:59:12.014351 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:12.014334    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 18 13:59:12.014453 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:12.014358    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 18 13:59:12.029360 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:12.029342    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2mgfl"
Sep 18 13:59:12.029450 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:12.029434    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2mgfl" podUID="57282ff1-fab4-4cd4-8087-5517bd249f10"
Sep 18 13:59:12.143054 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:12.143023    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnrrz" event={"ID":"1375ec1e-4106-47ba-867a-42a96ed1014c","Type":"ContainerStarted","Data":"79791e9a7f9b3619fe4e04bbd03baaeec3ac198b18ab687f22990c88ee401d0c"}
Sep 18 13:59:12.144743 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:12.144717    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-469c7" event={"ID":"7f65deed-0197-489a-a602-062486b50a57","Type":"ContainerStarted","Data":"a065123ac8a6fbd1e948550347a6d62a36f4582fb218a506d5f317a443dc48fd"}
Sep 18 13:59:12.164293 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:12.164244    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-469c7" podStartSLOduration=9.144700504 podStartE2EDuration="22.164228626s" totalImagesPullingTime="13.019528122s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:58:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:59:12.164142212 +0000 UTC m=+24.052858142" watchObservedRunningTime="2026-09-18 13:59:12.164228626 +0000 UTC m=+24.052944557"
Sep 18 13:59:12.338106 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:12.338070    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:12.441913 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:12.441886    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:12.720626 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:12.720557    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:12.828211 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:12.828155    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:12.904519 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:12.904496    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:13.029707 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:13.029686    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2hvmm"
Sep 18 13:59:13.029814 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:13.029687    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7msq"
Sep 18 13:59:13.029895 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:13.029864    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j7msq" podUID="7e8de495-7804-4c38-af57-655459217f1c"
Sep 18 13:59:13.029895 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:13.029773    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2hvmm" podUID="ca72836e-7ed0-47e5-a106-2f47340eb6b1"
Sep 18 13:59:13.151325 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:13.151305    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sg4tv_4626a667-4cd6-468c-8ef3-6e3a925c2608/ovn-acl-logging/0.log"
Sep 18 13:59:13.151791 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:13.151766    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv" event={"ID":"4626a667-4cd6-468c-8ef3-6e3a925c2608","Type":"ContainerStarted","Data":"0d9fdfe4f7b83f374e4d0826aec08c1f7c387bb06389123cc16dffe4e9e9eb3b"}
Sep 18 13:59:13.154161 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:13.154123    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnrrz" event={"ID":"1375ec1e-4106-47ba-867a-42a96ed1014c","Type":"ContainerStarted","Data":"6d729df1da6ab1113c28507add793458d819f1c8a6c60ef12648cf3972c2976b"}
Sep 18 13:59:13.179758 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:13.179710    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gnrrz" podStartSLOduration=3.729964555 podStartE2EDuration="23.179691534s" totalImagesPullingTime="19.449726979s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:58:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:59:13.17736078 +0000 UTC m=+25.066076712" watchObservedRunningTime="2026-09-18 13:59:13.179691534 +0000 UTC m=+25.068407465"
Sep 18 13:59:13.761911 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:13.761879    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-fkrgg"
Sep 18 13:59:13.762550 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:13.762532    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-fkrgg"
Sep 18 13:59:13.780462 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:13.780439    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:13.888293 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:13.888252    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:14.030154 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:14.030091    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2mgfl"
Sep 18 13:59:14.030288 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:14.030214    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2mgfl" podUID="57282ff1-fab4-4cd4-8087-5517bd249f10"
Sep 18 13:59:15.029480 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:15.029452    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7msq"
Sep 18 13:59:15.029931 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:15.029559    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j7msq" podUID="7e8de495-7804-4c38-af57-655459217f1c"
Sep 18 13:59:15.029931 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:15.029612    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2hvmm"
Sep 18 13:59:15.029931 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:15.029699    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2hvmm" podUID="ca72836e-7ed0-47e5-a106-2f47340eb6b1"
Sep 18 13:59:15.070255 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:15.070236    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t9ks5_690b430b-1984-4119-beae-85786c10b267/dns-node-resolver/0.log"
Sep 18 13:59:15.099288 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:15.099255    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:15.210563 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:15.210537    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:15.311231 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:15.311207    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:15.414450 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:15.414419    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:15.473668 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:15.473648    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8n79p_dbb463f3-8dcd-413d-859f-5dcf87c59115/node-ca/0.log"
Sep 18 13:59:16.030279 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:16.030048    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2mgfl"
Sep 18 13:59:16.030791 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:16.030334    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2mgfl" podUID="57282ff1-fab4-4cd4-8087-5517bd249f10"
Sep 18 13:59:16.156712 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:16.156686    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:16.162247 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:16.162217    2579 generic.go:356] "Generic (PLEG): container finished" podID="58a315d7-c375-4d9d-9b8b-fe148b1329e1" containerID="747da5d2dc149c92f062573b47c478a6c57909688fd83570285d86d55c5d11e4" exitCode=0
Sep 18 13:59:16.162334 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:16.162299    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p5f5h" event={"ID":"58a315d7-c375-4d9d-9b8b-fe148b1329e1","Type":"ContainerDied","Data":"747da5d2dc149c92f062573b47c478a6c57909688fd83570285d86d55c5d11e4"}
Sep 18 13:59:16.166760 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:16.166743    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sg4tv_4626a667-4cd6-468c-8ef3-6e3a925c2608/ovn-acl-logging/0.log"
Sep 18 13:59:16.167236 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:16.167204    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv" event={"ID":"4626a667-4cd6-468c-8ef3-6e3a925c2608","Type":"ContainerStarted","Data":"6f85830bd484e7d12170388a8b5d37161965108764049a7af3bb637674ad9f5b"}
Sep 18 13:59:16.167503 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:16.167469    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:59:16.167675 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:16.167651    2579 scope.go:117] "RemoveContainer" containerID="37a1a203681d866b3abbb049cbb00812a2da513d42777889300fc56b6640a2b5"
Sep 18 13:59:16.185216 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:16.185194    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:59:16.248288 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:16.248263    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:16.453887 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:16.453864    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:59:16.583184 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:16.583153    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-fkrgg"
Sep 18 13:59:16.583276 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:16.583263    2579 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 18 13:59:16.583909 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:16.583882    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-fkrgg"
Sep 18 13:59:16.659497 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:16.659446    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:16.945149 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:16.945077    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:17.029049 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:17.029023    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2hvmm"
Sep 18 13:59:17.029049 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:17.029053    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7msq"
Sep 18 13:59:17.029271 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:17.029141    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2hvmm" podUID="ca72836e-7ed0-47e5-a106-2f47340eb6b1"
Sep 18 13:59:17.029314 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:17.029294    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j7msq" podUID="7e8de495-7804-4c38-af57-655459217f1c"
Sep 18 13:59:17.055065 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:17.055036    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:17.173289 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:17.173260    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sg4tv_4626a667-4cd6-468c-8ef3-6e3a925c2608/ovn-acl-logging/0.log"
Sep 18 13:59:17.173649 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:17.173626    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv" event={"ID":"4626a667-4cd6-468c-8ef3-6e3a925c2608","Type":"ContainerStarted","Data":"c8f567e0e221bfb7cdb1d33fc107018f2567db9ba646eaec220a4cb5c7c74c64"}
Sep 18 13:59:17.173949 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:17.173935    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:59:17.188865 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:17.188842    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:59:17.207673 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:17.207578    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv" podStartSLOduration=9.908897636 podStartE2EDuration="27.207562451s" totalImagesPullingTime="17.298664815s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:58:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:59:17.20625271 +0000 UTC m=+29.094968640" watchObservedRunningTime="2026-09-18 13:59:17.207562451 +0000 UTC m=+29.096278360"
Sep 18 13:59:17.793810 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:17.793586    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2hvmm"]
Sep 18 13:59:17.793957 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:17.793825    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2hvmm"
Sep 18 13:59:17.793957 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:17.793919    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2hvmm" podUID="ca72836e-7ed0-47e5-a106-2f47340eb6b1"
Sep 18 13:59:17.797284 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:17.797259    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2mgfl"]
Sep 18 13:59:17.797420 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:17.797356    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2mgfl"
Sep 18 13:59:17.797507 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:17.797472    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2mgfl" podUID="57282ff1-fab4-4cd4-8087-5517bd249f10"
Sep 18 13:59:17.809334 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:17.809315    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j7msq"]
Sep 18 13:59:17.809444 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:17.809383    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7msq"
Sep 18 13:59:17.809501 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:17.809477    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j7msq" podUID="7e8de495-7804-4c38-af57-655459217f1c"
Sep 18 13:59:17.984509 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:17.984486    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:18.086993 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:18.086972    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:18.177853 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:18.177823    2579 generic.go:356] "Generic (PLEG): container finished" podID="58a315d7-c375-4d9d-9b8b-fe148b1329e1" containerID="1224b7de07c38445b91046d199d4706e65933109b850d9ed0eb04cc525ee5e5e" exitCode=0
Sep 18 13:59:18.177938 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:18.177850    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p5f5h" event={"ID":"58a315d7-c375-4d9d-9b8b-fe148b1329e1","Type":"ContainerDied","Data":"1224b7de07c38445b91046d199d4706e65933109b850d9ed0eb04cc525ee5e5e"}
Sep 18 13:59:19.029336 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:19.029308    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7msq"
Sep 18 13:59:19.029477 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:19.029400    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j7msq" podUID="7e8de495-7804-4c38-af57-655459217f1c"
Sep 18 13:59:19.394048 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:19.394021    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:19.495655 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:19.495635    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:19.860316 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:19.860286    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:19.961208 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:19.961157    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:20.030152 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:20.030132    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2hvmm"
Sep 18 13:59:20.030276 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:20.030231    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2hvmm" podUID="ca72836e-7ed0-47e5-a106-2f47340eb6b1"
Sep 18 13:59:20.030350 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:20.030301    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2mgfl"
Sep 18 13:59:20.030433 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:20.030411    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2mgfl" podUID="57282ff1-fab4-4cd4-8087-5517bd249f10"
Sep 18 13:59:20.113325 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:20.113304    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:20.177993 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:20.177946    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:20.183536 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:20.183517    2579 generic.go:356] "Generic (PLEG): container finished" podID="58a315d7-c375-4d9d-9b8b-fe148b1329e1" containerID="4a8ae57df9ccde209d45b20631175d3776a05932d7da4b1ffde9ee8fd781b695" exitCode=0
Sep 18 13:59:20.183614 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:20.183556    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p5f5h" event={"ID":"58a315d7-c375-4d9d-9b8b-fe148b1329e1","Type":"ContainerDied","Data":"4a8ae57df9ccde209d45b20631175d3776a05932d7da4b1ffde9ee8fd781b695"}
Sep 18 13:59:20.626154 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:20.626118    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:21.029351 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:21.029253    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7msq"
Sep 18 13:59:21.029508 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:21.029397    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j7msq" podUID="7e8de495-7804-4c38-af57-655459217f1c"
Sep 18 13:59:21.734638 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:21.734609    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:21.806240 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:21.806207    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:22.029914 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:22.029844    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2mgfl"
Sep 18 13:59:22.029914 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:22.029858    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2hvmm"
Sep 18 13:59:22.030111 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:22.029951    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2mgfl" podUID="57282ff1-fab4-4cd4-8087-5517bd249f10"
Sep 18 13:59:22.030111 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:22.030092    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2hvmm" podUID="ca72836e-7ed0-47e5-a106-2f47340eb6b1"
Sep 18 13:59:22.055457 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:22.055432    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:22.823236 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:22.823198    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:23.029699 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:23.029666    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7msq"
Sep 18 13:59:23.029839 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:23.029790    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j7msq" podUID="7e8de495-7804-4c38-af57-655459217f1c"
Sep 18 13:59:23.111705 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:23.111651    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-120.ec2.internal" event="NodeReady"
Sep 18 13:59:23.111819 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:23.111749    2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 18 13:59:23.651716 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:23.651682    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:23.659749 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:23.659720    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57282ff1-fab4-4cd4-8087-5517bd249f10-metrics-certs\") pod \"network-metrics-daemon-2mgfl\" (UID: \"57282ff1-fab4-4cd4-8087-5517bd249f10\") " pod="openshift-multus/network-metrics-daemon-2mgfl"
Sep 18 13:59:23.659895 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:23.659865    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:59:23.659958 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:23.659926    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57282ff1-fab4-4cd4-8087-5517bd249f10-metrics-certs podName:57282ff1-fab4-4cd4-8087-5517bd249f10 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:55.659911877 +0000 UTC m=+67.548627786 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57282ff1-fab4-4cd4-8087-5517bd249f10-metrics-certs") pod "network-metrics-daemon-2mgfl" (UID: "57282ff1-fab4-4cd4-8087-5517bd249f10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:59:23.760485 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:23.760458    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t8sk\" (UniqueName: \"kubernetes.io/projected/7e8de495-7804-4c38-af57-655459217f1c-kube-api-access-4t8sk\") pod \"network-check-target-j7msq\" (UID: \"7e8de495-7804-4c38-af57-655459217f1c\") " pod="openshift-network-diagnostics/network-check-target-j7msq"
Sep 18 13:59:23.760485 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:23.760489    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ca72836e-7ed0-47e5-a106-2f47340eb6b1-original-pull-secret\") pod \"global-pull-secret-syncer-2hvmm\" (UID: \"ca72836e-7ed0-47e5-a106-2f47340eb6b1\") " pod="kube-system/global-pull-secret-syncer-2hvmm"
Sep 18 13:59:23.760671 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:23.760597    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 13:59:23.760671 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:23.760613    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 13:59:23.760671 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:23.760633    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 13:59:23.760671 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:23.760645    2579 projected.go:196] Error preparing data for projected volume kube-api-access-4t8sk for pod openshift-network-diagnostics/network-check-target-j7msq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:59:23.760827 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:23.760645    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca72836e-7ed0-47e5-a106-2f47340eb6b1-original-pull-secret podName:ca72836e-7ed0-47e5-a106-2f47340eb6b1 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:55.760632867 +0000 UTC m=+67.649348781 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ca72836e-7ed0-47e5-a106-2f47340eb6b1-original-pull-secret") pod "global-pull-secret-syncer-2hvmm" (UID: "ca72836e-7ed0-47e5-a106-2f47340eb6b1") : object "kube-system"/"original-pull-secret" not registered
Sep 18 13:59:23.760827 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:23.760706    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7e8de495-7804-4c38-af57-655459217f1c-kube-api-access-4t8sk podName:7e8de495-7804-4c38-af57-655459217f1c nodeName:}" failed. No retries permitted until 2026-09-18 13:59:55.760690743 +0000 UTC m=+67.649406656 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-4t8sk" (UniqueName: "kubernetes.io/projected/7e8de495-7804-4c38-af57-655459217f1c-kube-api-access-4t8sk") pod "network-check-target-j7msq" (UID: "7e8de495-7804-4c38-af57-655459217f1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:59:23.765004 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:23.764975    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:23.866013 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:23.865986    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:23.964447 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:23.964340    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:24.029936 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:24.029910    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2hvmm"
Sep 18 13:59:24.030075 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:24.029959    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2mgfl"
Sep 18 13:59:24.032906 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:24.032539    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 13:59:24.036341 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:24.036320    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8cnh9\""
Sep 18 13:59:24.036664 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:24.036360    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 13:59:24.066606 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:24.066580    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:24.164595 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:24.164555    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:24.543965 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:24.543927    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:25.029526 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:25.029500    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7msq"
Sep 18 13:59:25.031998 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:25.031974    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 13:59:25.032123 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:25.032089    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-z6j5p\""
Sep 18 13:59:25.032123 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:25.032091    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 13:59:25.348580 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:25.348556    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:25.552713 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:25.552694    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:25.653981 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:25.653948    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:25.753926 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:25.753899    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:25.952992 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:25.952813    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:26.105316 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:26.105282    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:26.199356 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:26.199333    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:26.199544 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:26.199523    2579 generic.go:356] "Generic (PLEG): container finished" podID="58a315d7-c375-4d9d-9b8b-fe148b1329e1" containerID="748f7c43b6051a487f30904376601290c981cbae505ca2d4ebbc08c2ab249322" exitCode=0
Sep 18 13:59:26.199616 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:26.199589    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p5f5h" event={"ID":"58a315d7-c375-4d9d-9b8b-fe148b1329e1","Type":"ContainerDied","Data":"748f7c43b6051a487f30904376601290c981cbae505ca2d4ebbc08c2ab249322"}
Sep 18 13:59:26.957132 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:26.957100    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:27.204445 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:27.204416    2579 generic.go:356] "Generic (PLEG): container finished" podID="58a315d7-c375-4d9d-9b8b-fe148b1329e1" containerID="4c15b6ca4dfc9714b7263e72d2e7f8ccd595a17f3e52e958bef7c152d023866a" exitCode=0
Sep 18 13:59:27.204722 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:27.204472    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p5f5h" event={"ID":"58a315d7-c375-4d9d-9b8b-fe148b1329e1","Type":"ContainerDied","Data":"4c15b6ca4dfc9714b7263e72d2e7f8ccd595a17f3e52e958bef7c152d023866a"}
Sep 18 13:59:27.361457 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:27.361434    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:27.509197 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:27.509088    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:27.621878 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:27.621836    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:27.789432 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:27.789332    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:28.120782 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:28.120741    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:28.209940 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:28.209912    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p5f5h" event={"ID":"58a315d7-c375-4d9d-9b8b-fe148b1329e1","Type":"ContainerStarted","Data":"37e3b4ff3e947fd2569a4c8490b7740d39494bde206001c7d62239b096d51bf6"}
Sep 18 13:59:28.237609 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:28.237580    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:28.238210 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:28.238140    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-p5f5h" podStartSLOduration=9.325011457 podStartE2EDuration="38.238124992s" totalImagesPullingTime="28.913113535s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:58:50 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:59:28.237022084 +0000 UTC m=+40.125738015" watchObservedRunningTime="2026-09-18 13:59:28.238124992 +0000 UTC m=+40.126840947"
Sep 18 13:59:28.952082 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:28.952045    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:29.165056 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:29.165021    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:29.296714 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:29.296690    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:29.394998 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:29.394978    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:29.495386 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:29.495363    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:29.595936 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:29.595882    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:29.748436 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:29.748412    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:29.979234 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:29.979166    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:30.761714 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:30.761685    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:31.194396 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:31.194365    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:31.566214 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:31.566166    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:31.762201 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:31.762152    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:32.152359 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:32.152330    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:32.493907 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:32.493847    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:32.966705 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:32.966677    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:33.208186 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:33.208146    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:33.692073 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:33.692046    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:33.828118 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:33.828093    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:34.172215 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:34.172190    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:34.369058 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:34.369031    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:34.675938 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:34.675916    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:41.078630 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:41.078593    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:41.135593 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:41.135558    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:43.475166 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.475131    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:43.584351 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.584329    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:43.693195 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.693156    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-9b5c4599-q9jk2"]
Sep 18 13:59:43.718768 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.718739    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9b5c4599-q9jk2"
Sep 18 13:59:43.727158 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.727114    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-w5qd9\""
Sep 18 13:59:43.727776 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.727760    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 18 13:59:43.727823 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.727777    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 18 13:59:43.729949 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.729931    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 18 13:59:43.734832 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.734814    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-9b5c4599-q9jk2"]
Sep 18 13:59:43.743777 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.743757    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ddf856f5c-4wwwq"]
Sep 18 13:59:43.746381 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.746360    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 18 13:59:43.765991 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.765972    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74fcd9d577-4w4hq"]
Sep 18 13:59:43.766100 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.766085    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ddf856f5c-4wwwq"
Sep 18 13:59:43.768218 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.768198    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 18 13:59:43.768440 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.768420    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 18 13:59:43.769102 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.769081    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 18 13:59:43.769208 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.769083    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 18 13:59:43.786503 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.786487    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d8656f554-r9bff"]
Sep 18 13:59:43.786636 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.786620    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74fcd9d577-4w4hq"
Sep 18 13:59:43.795972 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.795956    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 18 13:59:43.796232 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.796215    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 18 13:59:43.796318 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.796217    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 18 13:59:43.796684 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.796668    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 18 13:59:43.802621 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.802602    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-9b5c4599-q9jk2"]
Sep 18 13:59:43.802703 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.802635    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ddf856f5c-4wwwq"]
Sep 18 13:59:43.802703 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.802651    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d8656f554-r9bff"]
Sep 18 13:59:43.802703 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.802660    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74fcd9d577-4w4hq"]
Sep 18 13:59:43.802795 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.802711    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d8656f554-r9bff"
Sep 18 13:59:43.802795 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:43.802740    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[bound-sa-token ca-trust-extracted image-registry-private-configuration installation-pull-secrets kube-api-access-lznxw registry-certificates registry-tls trusted-ca], unattached volumes=[], failed to process volumes=[bound-sa-token ca-trust-extracted image-registry-private-configuration installation-pull-secrets kube-api-access-lznxw registry-certificates registry-tls trusted-ca]: context canceled" pod="openshift-image-registry/image-registry-9b5c4599-q9jk2" podUID="2f629f41-2508-48d0-800b-d541840d2082"
Sep 18 13:59:43.805925 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.805908    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 18 13:59:43.806291 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.806276    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-7lmcx\""
Sep 18 13:59:43.818394 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.818373    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:43.825710 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.825691    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-c6gvb"]
Sep 18 13:59:43.847260 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.847243    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-549d599756-4kgjb"]
Sep 18 13:59:43.847367 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.847352    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c6gvb"
Sep 18 13:59:43.850845 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.850829    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 13:59:43.850968 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.850953    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fh88p\""
Sep 18 13:59:43.851195 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.851181    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 18 13:59:43.861780 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.861758    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-gjcx2"]
Sep 18 13:59:43.861861 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.861835    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-549d599756-4kgjb"
Sep 18 13:59:43.883012 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.882940    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gjcx2"]
Sep 18 13:59:43.883090 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.883031    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c6gvb"]
Sep 18 13:59:43.883090 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.883048    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-549d599756-4kgjb"]
Sep 18 13:59:43.883090 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.882995    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gjcx2"
Sep 18 13:59:43.887215 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.887198    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 18 13:59:43.887380 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.887362    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-5l9f5\""
Sep 18 13:59:43.887428 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.887391    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 13:59:43.888059 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.887914    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 18 13:59:43.894949 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.894930    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2f629f41-2508-48d0-800b-d541840d2082-bound-sa-token\") pod \"image-registry-9b5c4599-q9jk2\" (UID: \"2f629f41-2508-48d0-800b-d541840d2082\") " pod="openshift-image-registry/image-registry-9b5c4599-q9jk2"
Sep 18 13:59:43.895035 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.894957    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/fceb828f-1889-4dc2-8201-1e7a74d628bb-klusterlet-config\") pod \"klusterlet-addon-workmgr-5ddf856f5c-4wwwq\" (UID: \"fceb828f-1889-4dc2-8201-1e7a74d628bb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ddf856f5c-4wwwq"
Sep 18 13:59:43.895035 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.894976    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbe49c7e-f072-46a0-a0c1-47344a767afc-additional-service-ca\") pod \"cluster-proxy-proxy-agent-74fcd9d577-4w4hq\" (UID: \"fbe49c7e-f072-46a0-a0c1-47344a767afc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74fcd9d577-4w4hq"
Sep 18 13:59:43.895035 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.894997    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2f629f41-2508-48d0-800b-d541840d2082-installation-pull-secrets\") pod \"image-registry-9b5c4599-q9jk2\" (UID: \"2f629f41-2508-48d0-800b-d541840d2082\") " pod="openshift-image-registry/image-registry-9b5c4599-q9jk2"
Sep 18 13:59:43.895189 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.895038    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/fbe49c7e-f072-46a0-a0c1-47344a767afc-hub\") pod \"cluster-proxy-proxy-agent-74fcd9d577-4w4hq\" (UID: \"fbe49c7e-f072-46a0-a0c1-47344a767afc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74fcd9d577-4w4hq"
Sep 18 13:59:43.895189 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.895056    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zhd6\" (UniqueName: \"kubernetes.io/projected/fbe49c7e-f072-46a0-a0c1-47344a767afc-kube-api-access-6zhd6\") pod \"cluster-proxy-proxy-agent-74fcd9d577-4w4hq\" (UID: \"fbe49c7e-f072-46a0-a0c1-47344a767afc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74fcd9d577-4w4hq"
Sep 18 13:59:43.895189 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.895071    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2f629f41-2508-48d0-800b-d541840d2082-registry-tls\") pod \"image-registry-9b5c4599-q9jk2\" (UID: \"2f629f41-2508-48d0-800b-d541840d2082\") " pod="openshift-image-registry/image-registry-9b5c4599-q9jk2"
Sep 18 13:59:43.895189 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.895089    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2f629f41-2508-48d0-800b-d541840d2082-ca-trust-extracted\") pod \"image-registry-9b5c4599-q9jk2\" (UID: \"2f629f41-2508-48d0-800b-d541840d2082\") " pod="openshift-image-registry/image-registry-9b5c4599-q9jk2"
Sep 18 13:59:43.895377 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.895188    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2f629f41-2508-48d0-800b-d541840d2082-trusted-ca\") pod \"image-registry-9b5c4599-q9jk2\" (UID: \"2f629f41-2508-48d0-800b-d541840d2082\") " pod="openshift-image-registry/image-registry-9b5c4599-q9jk2"
Sep 18 13:59:43.895377 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.895221    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fbe49c7e-f072-46a0-a0c1-47344a767afc-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-74fcd9d577-4w4hq\" (UID: \"fbe49c7e-f072-46a0-a0c1-47344a767afc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74fcd9d577-4w4hq"
Sep 18 13:59:43.895377 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.895266    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/fbe49c7e-f072-46a0-a0c1-47344a767afc-ca\") pod \"cluster-proxy-proxy-agent-74fcd9d577-4w4hq\" (UID: \"fbe49c7e-f072-46a0-a0c1-47344a767afc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74fcd9d577-4w4hq"
Sep 18 13:59:43.895377 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.895290    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/fbe49c7e-f072-46a0-a0c1-47344a767afc-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-74fcd9d577-4w4hq\" (UID: \"fbe49c7e-f072-46a0-a0c1-47344a767afc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74fcd9d577-4w4hq"
Sep 18 13:59:43.895377 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.895314    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fceb828f-1889-4dc2-8201-1e7a74d628bb-tmp\") pod \"klusterlet-addon-workmgr-5ddf856f5c-4wwwq\" (UID: \"fceb828f-1889-4dc2-8201-1e7a74d628bb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ddf856f5c-4wwwq"
Sep 18 13:59:43.895377 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.895343    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2f629f41-2508-48d0-800b-d541840d2082-image-registry-private-configuration\") pod \"image-registry-9b5c4599-q9jk2\" (UID: \"2f629f41-2508-48d0-800b-d541840d2082\") " pod="openshift-image-registry/image-registry-9b5c4599-q9jk2"
Sep 18 13:59:43.895604 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.895395    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/da8ad0db-9a94-49d8-b301-42e2514bbb46-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-d8656f554-r9bff\" (UID: \"da8ad0db-9a94-49d8-b301-42e2514bbb46\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d8656f554-r9bff"
Sep 18 13:59:43.895604 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.895414    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2f629f41-2508-48d0-800b-d541840d2082-registry-certificates\") pod \"image-registry-9b5c4599-q9jk2\" (UID: \"2f629f41-2508-48d0-800b-d541840d2082\") " pod="openshift-image-registry/image-registry-9b5c4599-q9jk2"
Sep 18 13:59:43.895604 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.895428    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lznxw\" (UniqueName: \"kubernetes.io/projected/2f629f41-2508-48d0-800b-d541840d2082-kube-api-access-lznxw\") pod \"image-registry-9b5c4599-q9jk2\" (UID: \"2f629f41-2508-48d0-800b-d541840d2082\") " pod="openshift-image-registry/image-registry-9b5c4599-q9jk2"
Sep 18 13:59:43.895604 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.895444    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dsqz\" (UniqueName: \"kubernetes.io/projected/fceb828f-1889-4dc2-8201-1e7a74d628bb-kube-api-access-4dsqz\") pod \"klusterlet-addon-workmgr-5ddf856f5c-4wwwq\" (UID: \"fceb828f-1889-4dc2-8201-1e7a74d628bb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ddf856f5c-4wwwq"
Sep 18 13:59:43.895604 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.895491    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrgjq\" (UniqueName: \"kubernetes.io/projected/da8ad0db-9a94-49d8-b301-42e2514bbb46-kube-api-access-rrgjq\") pod \"managed-serviceaccount-addon-agent-d8656f554-r9bff\" (UID: \"da8ad0db-9a94-49d8-b301-42e2514bbb46\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d8656f554-r9bff"
Sep 18 13:59:43.945977 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.945954    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-52cv4"]
Sep 18 13:59:43.964837 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.964817    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-52cv4"
Sep 18 13:59:43.968595 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.968573    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 18 13:59:43.968595 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.968586    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 18 13:59:43.968767 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.968667    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-52cv4"]
Sep 18 13:59:43.969140 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.969117    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p5mbv\""
Sep 18 13:59:43.969359 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.969343    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 13:59:43.971720 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.971704    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 18 13:59:43.993439 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.993395    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:43.996443 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.996417    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2f629f41-2508-48d0-800b-d541840d2082-trusted-ca\") pod \"image-registry-9b5c4599-q9jk2\" (UID: \"2f629f41-2508-48d0-800b-d541840d2082\") " pod="openshift-image-registry/image-registry-9b5c4599-q9jk2"
Sep 18 13:59:43.996505 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.996452    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fbe49c7e-f072-46a0-a0c1-47344a767afc-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-74fcd9d577-4w4hq\" (UID: \"fbe49c7e-f072-46a0-a0c1-47344a767afc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74fcd9d577-4w4hq"
Sep 18 13:59:43.996505 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.996468    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/fbe49c7e-f072-46a0-a0c1-47344a767afc-ca\") pod \"cluster-proxy-proxy-agent-74fcd9d577-4w4hq\" (UID: \"fbe49c7e-f072-46a0-a0c1-47344a767afc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74fcd9d577-4w4hq"
Sep 18 13:59:43.996596 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.996580    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/fbe49c7e-f072-46a0-a0c1-47344a767afc-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-74fcd9d577-4w4hq\" (UID: \"fbe49c7e-f072-46a0-a0c1-47344a767afc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74fcd9d577-4w4hq"
Sep 18 13:59:43.996634 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.996608    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fceb828f-1889-4dc2-8201-1e7a74d628bb-tmp\") pod \"klusterlet-addon-workmgr-5ddf856f5c-4wwwq\" (UID: \"fceb828f-1889-4dc2-8201-1e7a74d628bb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ddf856f5c-4wwwq"
Sep 18 13:59:43.996634 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.996626    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2f629f41-2508-48d0-800b-d541840d2082-image-registry-private-configuration\") pod \"image-registry-9b5c4599-q9jk2\" (UID: \"2f629f41-2508-48d0-800b-d541840d2082\") " pod="openshift-image-registry/image-registry-9b5c4599-q9jk2"
Sep 18 13:59:43.996801 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.996644    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5bedf54a-fdc9-42b4-a3bb-df8621dd58cb-config-volume\") pod \"dns-default-c6gvb\" (UID: \"5bedf54a-fdc9-42b4-a3bb-df8621dd58cb\") " pod="openshift-dns/dns-default-c6gvb"
Sep 18 13:59:43.996801 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.996663    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ccb6a3bc-51ff-46d8-8a67-a97bd76f618c-registry-certificates\") pod \"image-registry-549d599756-4kgjb\" (UID: \"ccb6a3bc-51ff-46d8-8a67-a97bd76f618c\") " pod="openshift-image-registry/image-registry-549d599756-4kgjb"
Sep 18 13:59:43.996958 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.996798    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/da8ad0db-9a94-49d8-b301-42e2514bbb46-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-d8656f554-r9bff\" (UID: \"da8ad0db-9a94-49d8-b301-42e2514bbb46\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d8656f554-r9bff"
Sep 18 13:59:43.996958 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.996830    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7903d08d-219b-453f-9a68-3710e004f0e4-cert\") pod \"ingress-canary-gjcx2\" (UID: \"7903d08d-219b-453f-9a68-3710e004f0e4\") " pod="openshift-ingress-canary/ingress-canary-gjcx2"
Sep 18 13:59:43.996958 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.996871    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ccb6a3bc-51ff-46d8-8a67-a97bd76f618c-trusted-ca\") pod \"image-registry-549d599756-4kgjb\" (UID: \"ccb6a3bc-51ff-46d8-8a67-a97bd76f618c\") " pod="openshift-image-registry/image-registry-549d599756-4kgjb"
Sep 18 13:59:43.996958 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.996893    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dsqz\" (UniqueName: \"kubernetes.io/projected/fceb828f-1889-4dc2-8201-1e7a74d628bb-kube-api-access-4dsqz\") pod \"klusterlet-addon-workmgr-5ddf856f5c-4wwwq\" (UID: \"fceb828f-1889-4dc2-8201-1e7a74d628bb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ddf856f5c-4wwwq"
Sep 18 13:59:43.996958 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.996914    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ccb6a3bc-51ff-46d8-8a67-a97bd76f618c-ca-trust-extracted\") pod \"image-registry-549d599756-4kgjb\" (UID: \"ccb6a3bc-51ff-46d8-8a67-a97bd76f618c\") " pod="openshift-image-registry/image-registry-549d599756-4kgjb"
Sep 18 13:59:43.996958 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.996942    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fceb828f-1889-4dc2-8201-1e7a74d628bb-tmp\") pod \"klusterlet-addon-workmgr-5ddf856f5c-4wwwq\" (UID: \"fceb828f-1889-4dc2-8201-1e7a74d628bb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ddf856f5c-4wwwq"
Sep 18 13:59:43.996958 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.996955    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrgjq\" (UniqueName: \"kubernetes.io/projected/da8ad0db-9a94-49d8-b301-42e2514bbb46-kube-api-access-rrgjq\") pod \"managed-serviceaccount-addon-agent-d8656f554-r9bff\" (UID: \"da8ad0db-9a94-49d8-b301-42e2514bbb46\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d8656f554-r9bff"
Sep 18 13:59:43.997305 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.996976    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ccb6a3bc-51ff-46d8-8a67-a97bd76f618c-bound-sa-token\") pod \"image-registry-549d599756-4kgjb\" (UID: \"ccb6a3bc-51ff-46d8-8a67-a97bd76f618c\") " pod="openshift-image-registry/image-registry-549d599756-4kgjb"
Sep 18 13:59:43.997305 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.997017    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/fceb828f-1889-4dc2-8201-1e7a74d628bb-klusterlet-config\") pod \"klusterlet-addon-workmgr-5ddf856f5c-4wwwq\" (UID: \"fceb828f-1889-4dc2-8201-1e7a74d628bb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ddf856f5c-4wwwq"
Sep 18 13:59:43.997305 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.997045    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ccb6a3bc-51ff-46d8-8a67-a97bd76f618c-image-registry-private-configuration\") pod \"image-registry-549d599756-4kgjb\" (UID: \"ccb6a3bc-51ff-46d8-8a67-a97bd76f618c\") " pod="openshift-image-registry/image-registry-549d599756-4kgjb"
Sep 18 13:59:43.997305 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.997095    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbe49c7e-f072-46a0-a0c1-47344a767afc-additional-service-ca\") pod \"cluster-proxy-proxy-agent-74fcd9d577-4w4hq\" (UID: \"fbe49c7e-f072-46a0-a0c1-47344a767afc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74fcd9d577-4w4hq"
Sep 18 13:59:43.997305 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.997138    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/fbe49c7e-f072-46a0-a0c1-47344a767afc-hub\") pod \"cluster-proxy-proxy-agent-74fcd9d577-4w4hq\" (UID: \"fbe49c7e-f072-46a0-a0c1-47344a767afc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74fcd9d577-4w4hq"
Sep 18 13:59:43.997305 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.997189    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zhd6\" (UniqueName: \"kubernetes.io/projected/fbe49c7e-f072-46a0-a0c1-47344a767afc-kube-api-access-6zhd6\") pod \"cluster-proxy-proxy-agent-74fcd9d577-4w4hq\" (UID: \"fbe49c7e-f072-46a0-a0c1-47344a767afc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74fcd9d577-4w4hq"
Sep 18 13:59:43.997305 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.997218    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhl87\" (UniqueName: \"kubernetes.io/projected/5bedf54a-fdc9-42b4-a3bb-df8621dd58cb-kube-api-access-fhl87\") pod \"dns-default-c6gvb\" (UID: \"5bedf54a-fdc9-42b4-a3bb-df8621dd58cb\") " pod="openshift-dns/dns-default-c6gvb"
Sep 18 13:59:43.997305 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.997252    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr9tb\" (UniqueName: \"kubernetes.io/projected/7903d08d-219b-453f-9a68-3710e004f0e4-kube-api-access-zr9tb\") pod \"ingress-canary-gjcx2\" (UID: \"7903d08d-219b-453f-9a68-3710e004f0e4\") " pod="openshift-ingress-canary/ingress-canary-gjcx2"
Sep 18 13:59:43.997305 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.997272    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ccb6a3bc-51ff-46d8-8a67-a97bd76f618c-registry-tls\") pod \"image-registry-549d599756-4kgjb\" (UID: \"ccb6a3bc-51ff-46d8-8a67-a97bd76f618c\") " pod="openshift-image-registry/image-registry-549d599756-4kgjb"
Sep 18 13:59:43.997305 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.997292    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ccb6a3bc-51ff-46d8-8a67-a97bd76f618c-installation-pull-secrets\") pod \"image-registry-549d599756-4kgjb\" (UID: \"ccb6a3bc-51ff-46d8-8a67-a97bd76f618c\") " pod="openshift-image-registry/image-registry-549d599756-4kgjb"
Sep 18 13:59:43.997699 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.997428    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2f629f41-2508-48d0-800b-d541840d2082-registry-certificates\") pod \"image-registry-9b5c4599-q9jk2\" (UID: \"2f629f41-2508-48d0-800b-d541840d2082\") " pod="openshift-image-registry/image-registry-9b5c4599-q9jk2"
Sep 18 13:59:43.997699 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.997450    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lznxw\" (UniqueName: \"kubernetes.io/projected/2f629f41-2508-48d0-800b-d541840d2082-kube-api-access-lznxw\") pod \"image-registry-9b5c4599-q9jk2\" (UID: \"2f629f41-2508-48d0-800b-d541840d2082\") " pod="openshift-image-registry/image-registry-9b5c4599-q9jk2"
Sep 18 13:59:43.997699 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.997479    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2f629f41-2508-48d0-800b-d541840d2082-trusted-ca\") pod \"image-registry-9b5c4599-q9jk2\" (UID: \"2f629f41-2508-48d0-800b-d541840d2082\") " pod="openshift-image-registry/image-registry-9b5c4599-q9jk2"
Sep 18 13:59:43.997699 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.997568    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bedf54a-fdc9-42b4-a3bb-df8621dd58cb-tmp-dir\") pod \"dns-default-c6gvb\" (UID: \"5bedf54a-fdc9-42b4-a3bb-df8621dd58cb\") " pod="openshift-dns/dns-default-c6gvb"
Sep 18 13:59:43.997699 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.997590    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vrld\" (UniqueName: \"kubernetes.io/projected/ccb6a3bc-51ff-46d8-8a67-a97bd76f618c-kube-api-access-8vrld\") pod \"image-registry-549d599756-4kgjb\" (UID: \"ccb6a3bc-51ff-46d8-8a67-a97bd76f618c\") " pod="openshift-image-registry/image-registry-549d599756-4kgjb"
Sep 18 13:59:43.997932 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.997612    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2f629f41-2508-48d0-800b-d541840d2082-bound-sa-token\") pod \"image-registry-9b5c4599-q9jk2\" (UID: \"2f629f41-2508-48d0-800b-d541840d2082\") " pod="openshift-image-registry/image-registry-9b5c4599-q9jk2"
Sep 18 13:59:43.997932 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.997778    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5bedf54a-fdc9-42b4-a3bb-df8621dd58cb-metrics-tls\") pod \"dns-default-c6gvb\" (UID: \"5bedf54a-fdc9-42b4-a3bb-df8621dd58cb\") " pod="openshift-dns/dns-default-c6gvb"
Sep 18 13:59:43.997932 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.997820    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2f629f41-2508-48d0-800b-d541840d2082-installation-pull-secrets\") pod \"image-registry-9b5c4599-q9jk2\" (UID: \"2f629f41-2508-48d0-800b-d541840d2082\") " pod="openshift-image-registry/image-registry-9b5c4599-q9jk2"
Sep 18 13:59:43.997932 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.997881    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2f629f41-2508-48d0-800b-d541840d2082-registry-tls\") pod \"image-registry-9b5c4599-q9jk2\" (UID: \"2f629f41-2508-48d0-800b-d541840d2082\") " pod="openshift-image-registry/image-registry-9b5c4599-q9jk2"
Sep 18 13:59:43.997932 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.997914    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2f629f41-2508-48d0-800b-d541840d2082-ca-trust-extracted\") pod \"image-registry-9b5c4599-q9jk2\" (UID: \"2f629f41-2508-48d0-800b-d541840d2082\") " pod="openshift-image-registry/image-registry-9b5c4599-q9jk2"
Sep 18 13:59:43.998190 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.998051    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbe49c7e-f072-46a0-a0c1-47344a767afc-additional-service-ca\") pod \"cluster-proxy-proxy-agent-74fcd9d577-4w4hq\" (UID: \"fbe49c7e-f072-46a0-a0c1-47344a767afc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74fcd9d577-4w4hq"
Sep 18 13:59:43.998722 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.998460    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2f629f41-2508-48d0-800b-d541840d2082-registry-certificates\") pod \"image-registry-9b5c4599-q9jk2\" (UID: \"2f629f41-2508-48d0-800b-d541840d2082\") " pod="openshift-image-registry/image-registry-9b5c4599-q9jk2"
Sep 18 13:59:43.998808 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:43.998740    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2f629f41-2508-48d0-800b-d541840d2082-ca-trust-extracted\") pod \"image-registry-9b5c4599-q9jk2\" (UID: \"2f629f41-2508-48d0-800b-d541840d2082\") " pod="openshift-image-registry/image-registry-9b5c4599-q9jk2"
Sep 18 13:59:44.001729 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.001705    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/fbe49c7e-f072-46a0-a0c1-47344a767afc-hub\") pod \"cluster-proxy-proxy-agent-74fcd9d577-4w4hq\" (UID: \"fbe49c7e-f072-46a0-a0c1-47344a767afc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74fcd9d577-4w4hq"
Sep 18 13:59:44.002066 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.002047    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2f629f41-2508-48d0-800b-d541840d2082-installation-pull-secrets\") pod \"image-registry-9b5c4599-q9jk2\" (UID: \"2f629f41-2508-48d0-800b-d541840d2082\") " pod="openshift-image-registry/image-registry-9b5c4599-q9jk2"
Sep 18 13:59:44.002149 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.002076    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2f629f41-2508-48d0-800b-d541840d2082-image-registry-private-configuration\") pod \"image-registry-9b5c4599-q9jk2\" (UID: \"2f629f41-2508-48d0-800b-d541840d2082\") " pod="openshift-image-registry/image-registry-9b5c4599-q9jk2"
Sep 18 13:59:44.002231 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.002144    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/fbe49c7e-f072-46a0-a0c1-47344a767afc-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-74fcd9d577-4w4hq\" (UID: \"fbe49c7e-f072-46a0-a0c1-47344a767afc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74fcd9d577-4w4hq"
Sep 18 13:59:44.002284 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.002263    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/fceb828f-1889-4dc2-8201-1e7a74d628bb-klusterlet-config\") pod \"klusterlet-addon-workmgr-5ddf856f5c-4wwwq\" (UID: \"fceb828f-1889-4dc2-8201-1e7a74d628bb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ddf856f5c-4wwwq"
Sep 18 13:59:44.002429 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.002415    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2f629f41-2508-48d0-800b-d541840d2082-registry-tls\") pod \"image-registry-9b5c4599-q9jk2\" (UID: \"2f629f41-2508-48d0-800b-d541840d2082\") " pod="openshift-image-registry/image-registry-9b5c4599-q9jk2"
Sep 18 13:59:44.002464 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.002452    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/da8ad0db-9a94-49d8-b301-42e2514bbb46-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-d8656f554-r9bff\" (UID: \"da8ad0db-9a94-49d8-b301-42e2514bbb46\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d8656f554-r9bff"
Sep 18 13:59:44.002674 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.002656    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fbe49c7e-f072-46a0-a0c1-47344a767afc-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-74fcd9d577-4w4hq\" (UID: \"fbe49c7e-f072-46a0-a0c1-47344a767afc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74fcd9d577-4w4hq"
Sep 18 13:59:44.017751 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.017733    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/fbe49c7e-f072-46a0-a0c1-47344a767afc-ca\") pod \"cluster-proxy-proxy-agent-74fcd9d577-4w4hq\" (UID: \"fbe49c7e-f072-46a0-a0c1-47344a767afc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74fcd9d577-4w4hq"
Sep 18 13:59:44.035609 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.035582    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zhd6\" (UniqueName: \"kubernetes.io/projected/fbe49c7e-f072-46a0-a0c1-47344a767afc-kube-api-access-6zhd6\") pod \"cluster-proxy-proxy-agent-74fcd9d577-4w4hq\" (UID: \"fbe49c7e-f072-46a0-a0c1-47344a767afc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74fcd9d577-4w4hq"
Sep 18 13:59:44.035688 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.035652    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrgjq\" (UniqueName: \"kubernetes.io/projected/da8ad0db-9a94-49d8-b301-42e2514bbb46-kube-api-access-rrgjq\") pod \"managed-serviceaccount-addon-agent-d8656f554-r9bff\" (UID: \"da8ad0db-9a94-49d8-b301-42e2514bbb46\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d8656f554-r9bff"
Sep 18 13:59:44.035864 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.035849    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dsqz\" (UniqueName: \"kubernetes.io/projected/fceb828f-1889-4dc2-8201-1e7a74d628bb-kube-api-access-4dsqz\") pod \"klusterlet-addon-workmgr-5ddf856f5c-4wwwq\" (UID: \"fceb828f-1889-4dc2-8201-1e7a74d628bb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ddf856f5c-4wwwq"
Sep 18 13:59:44.036015 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.035997    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2f629f41-2508-48d0-800b-d541840d2082-bound-sa-token\") pod \"image-registry-9b5c4599-q9jk2\" (UID: \"2f629f41-2508-48d0-800b-d541840d2082\") " pod="openshift-image-registry/image-registry-9b5c4599-q9jk2"
Sep 18 13:59:44.036088 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.036075    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lznxw\" (UniqueName: \"kubernetes.io/projected/2f629f41-2508-48d0-800b-d541840d2082-kube-api-access-lznxw\") pod \"image-registry-9b5c4599-q9jk2\" (UID: \"2f629f41-2508-48d0-800b-d541840d2082\") " pod="openshift-image-registry/image-registry-9b5c4599-q9jk2"
Sep 18 13:59:44.075005 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.074987    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ddf856f5c-4wwwq"
Sep 18 13:59:44.099196 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.099162    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bedf54a-fdc9-42b4-a3bb-df8621dd58cb-tmp-dir\") pod \"dns-default-c6gvb\" (UID: \"5bedf54a-fdc9-42b4-a3bb-df8621dd58cb\") " pod="openshift-dns/dns-default-c6gvb"
Sep 18 13:59:44.099269 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.099202    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vrld\" (UniqueName: \"kubernetes.io/projected/ccb6a3bc-51ff-46d8-8a67-a97bd76f618c-kube-api-access-8vrld\") pod \"image-registry-549d599756-4kgjb\" (UID: \"ccb6a3bc-51ff-46d8-8a67-a97bd76f618c\") " pod="openshift-image-registry/image-registry-549d599756-4kgjb"
Sep 18 13:59:44.099269 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.099225    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t55zn\" (UniqueName: \"kubernetes.io/projected/5ac8c4ef-13cb-43f0-b07b-5540874267d3-kube-api-access-t55zn\") pod \"insights-runtime-extractor-52cv4\" (UID: \"5ac8c4ef-13cb-43f0-b07b-5540874267d3\") " pod="openshift-insights/insights-runtime-extractor-52cv4"
Sep 18 13:59:44.099357 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.099340    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5bedf54a-fdc9-42b4-a3bb-df8621dd58cb-metrics-tls\") pod \"dns-default-c6gvb\" (UID: \"5bedf54a-fdc9-42b4-a3bb-df8621dd58cb\") " pod="openshift-dns/dns-default-c6gvb"
Sep 18 13:59:44.099398 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.099385    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5ac8c4ef-13cb-43f0-b07b-5540874267d3-crio-socket\") pod \"insights-runtime-extractor-52cv4\" (UID: \"5ac8c4ef-13cb-43f0-b07b-5540874267d3\") " pod="openshift-insights/insights-runtime-extractor-52cv4"
Sep 18 13:59:44.099437 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.099418    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5bedf54a-fdc9-42b4-a3bb-df8621dd58cb-config-volume\") pod \"dns-default-c6gvb\" (UID: \"5bedf54a-fdc9-42b4-a3bb-df8621dd58cb\") " pod="openshift-dns/dns-default-c6gvb"
Sep 18 13:59:44.099487 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.099434    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ccb6a3bc-51ff-46d8-8a67-a97bd76f618c-registry-certificates\") pod \"image-registry-549d599756-4kgjb\" (UID: \"ccb6a3bc-51ff-46d8-8a67-a97bd76f618c\") " pod="openshift-image-registry/image-registry-549d599756-4kgjb"
Sep 18 13:59:44.099487 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.099451    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7903d08d-219b-453f-9a68-3710e004f0e4-cert\") pod \"ingress-canary-gjcx2\" (UID: \"7903d08d-219b-453f-9a68-3710e004f0e4\") " pod="openshift-ingress-canary/ingress-canary-gjcx2"
Sep 18 13:59:44.099487 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.099453    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bedf54a-fdc9-42b4-a3bb-df8621dd58cb-tmp-dir\") pod \"dns-default-c6gvb\" (UID: \"5bedf54a-fdc9-42b4-a3bb-df8621dd58cb\") " pod="openshift-dns/dns-default-c6gvb"
Sep 18 13:59:44.099487 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.099464    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ccb6a3bc-51ff-46d8-8a67-a97bd76f618c-trusted-ca\") pod \"image-registry-549d599756-4kgjb\" (UID: \"ccb6a3bc-51ff-46d8-8a67-a97bd76f618c\") " pod="openshift-image-registry/image-registry-549d599756-4kgjb"
Sep 18 13:59:44.099690 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.099494    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ccb6a3bc-51ff-46d8-8a67-a97bd76f618c-ca-trust-extracted\") pod \"image-registry-549d599756-4kgjb\" (UID: \"ccb6a3bc-51ff-46d8-8a67-a97bd76f618c\") " pod="openshift-image-registry/image-registry-549d599756-4kgjb"
Sep 18 13:59:44.099690 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.099522    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ccb6a3bc-51ff-46d8-8a67-a97bd76f618c-bound-sa-token\") pod \"image-registry-549d599756-4kgjb\" (UID: \"ccb6a3bc-51ff-46d8-8a67-a97bd76f618c\") " pod="openshift-image-registry/image-registry-549d599756-4kgjb"
Sep 18 13:59:44.099690 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.099550    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5ac8c4ef-13cb-43f0-b07b-5540874267d3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-52cv4\" (UID: \"5ac8c4ef-13cb-43f0-b07b-5540874267d3\") " pod="openshift-insights/insights-runtime-extractor-52cv4"
Sep 18 13:59:44.099690 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.099582    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ccb6a3bc-51ff-46d8-8a67-a97bd76f618c-image-registry-private-configuration\") pod \"image-registry-549d599756-4kgjb\" (UID: \"ccb6a3bc-51ff-46d8-8a67-a97bd76f618c\") " pod="openshift-image-registry/image-registry-549d599756-4kgjb"
Sep 18 13:59:44.099690 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.099625    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5ac8c4ef-13cb-43f0-b07b-5540874267d3-data-volume\") pod \"insights-runtime-extractor-52cv4\" (UID: \"5ac8c4ef-13cb-43f0-b07b-5540874267d3\") " pod="openshift-insights/insights-runtime-extractor-52cv4"
Sep 18 13:59:44.099690 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.099652    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5ac8c4ef-13cb-43f0-b07b-5540874267d3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-52cv4\" (UID: \"5ac8c4ef-13cb-43f0-b07b-5540874267d3\") " pod="openshift-insights/insights-runtime-extractor-52cv4"
Sep 18 13:59:44.099690 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.099680    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhl87\" (UniqueName: \"kubernetes.io/projected/5bedf54a-fdc9-42b4-a3bb-df8621dd58cb-kube-api-access-fhl87\") pod \"dns-default-c6gvb\" (UID: \"5bedf54a-fdc9-42b4-a3bb-df8621dd58cb\") " pod="openshift-dns/dns-default-c6gvb"
Sep 18 13:59:44.100027 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.099705    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr9tb\" (UniqueName: \"kubernetes.io/projected/7903d08d-219b-453f-9a68-3710e004f0e4-kube-api-access-zr9tb\") pod \"ingress-canary-gjcx2\" (UID: \"7903d08d-219b-453f-9a68-3710e004f0e4\") " pod="openshift-ingress-canary/ingress-canary-gjcx2"
Sep 18 13:59:44.100027 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.099727    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ccb6a3bc-51ff-46d8-8a67-a97bd76f618c-registry-tls\") pod \"image-registry-549d599756-4kgjb\" (UID: \"ccb6a3bc-51ff-46d8-8a67-a97bd76f618c\") " pod="openshift-image-registry/image-registry-549d599756-4kgjb"
Sep 18 13:59:44.100027 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.099751    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ccb6a3bc-51ff-46d8-8a67-a97bd76f618c-installation-pull-secrets\") pod \"image-registry-549d599756-4kgjb\" (UID: \"ccb6a3bc-51ff-46d8-8a67-a97bd76f618c\") " pod="openshift-image-registry/image-registry-549d599756-4kgjb"
Sep 18 13:59:44.100311 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.100294    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ccb6a3bc-51ff-46d8-8a67-a97bd76f618c-registry-certificates\") pod \"image-registry-549d599756-4kgjb\" (UID: \"ccb6a3bc-51ff-46d8-8a67-a97bd76f618c\") " pod="openshift-image-registry/image-registry-549d599756-4kgjb"
Sep 18 13:59:44.100384 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.100306    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ccb6a3bc-51ff-46d8-8a67-a97bd76f618c-ca-trust-extracted\") pod \"image-registry-549d599756-4kgjb\" (UID: \"ccb6a3bc-51ff-46d8-8a67-a97bd76f618c\") " pod="openshift-image-registry/image-registry-549d599756-4kgjb"
Sep 18 13:59:44.100524 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.100484    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ccb6a3bc-51ff-46d8-8a67-a97bd76f618c-trusted-ca\") pod \"image-registry-549d599756-4kgjb\" (UID: \"ccb6a3bc-51ff-46d8-8a67-a97bd76f618c\") " pod="openshift-image-registry/image-registry-549d599756-4kgjb"
Sep 18 13:59:44.100524 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.100510    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5bedf54a-fdc9-42b4-a3bb-df8621dd58cb-config-volume\") pod \"dns-default-c6gvb\" (UID: \"5bedf54a-fdc9-42b4-a3bb-df8621dd58cb\") " pod="openshift-dns/dns-default-c6gvb"
Sep 18 13:59:44.101874 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.101848    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5bedf54a-fdc9-42b4-a3bb-df8621dd58cb-metrics-tls\") pod \"dns-default-c6gvb\" (UID: \"5bedf54a-fdc9-42b4-a3bb-df8621dd58cb\") " pod="openshift-dns/dns-default-c6gvb"
Sep 18 13:59:44.101960 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.101935    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7903d08d-219b-453f-9a68-3710e004f0e4-cert\") pod \"ingress-canary-gjcx2\" (UID: \"7903d08d-219b-453f-9a68-3710e004f0e4\") " pod="openshift-ingress-canary/ingress-canary-gjcx2"
Sep 18 13:59:44.102276 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.102258    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ccb6a3bc-51ff-46d8-8a67-a97bd76f618c-installation-pull-secrets\") pod \"image-registry-549d599756-4kgjb\" (UID: \"ccb6a3bc-51ff-46d8-8a67-a97bd76f618c\") " pod="openshift-image-registry/image-registry-549d599756-4kgjb"
Sep 18 13:59:44.102585 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.102563    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ccb6a3bc-51ff-46d8-8a67-a97bd76f618c-image-registry-private-configuration\") pod \"image-registry-549d599756-4kgjb\" (UID: \"ccb6a3bc-51ff-46d8-8a67-a97bd76f618c\") " pod="openshift-image-registry/image-registry-549d599756-4kgjb"
Sep 18 13:59:44.102659 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.102569    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ccb6a3bc-51ff-46d8-8a67-a97bd76f618c-registry-tls\") pod \"image-registry-549d599756-4kgjb\" (UID: \"ccb6a3bc-51ff-46d8-8a67-a97bd76f618c\") " pod="openshift-image-registry/image-registry-549d599756-4kgjb"
Sep 18 13:59:44.104309 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.104296    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74fcd9d577-4w4hq"
Sep 18 13:59:44.110686 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.110668    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d8656f554-r9bff"
Sep 18 13:59:44.115341 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.115322    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhl87\" (UniqueName: \"kubernetes.io/projected/5bedf54a-fdc9-42b4-a3bb-df8621dd58cb-kube-api-access-fhl87\") pod \"dns-default-c6gvb\" (UID: \"5bedf54a-fdc9-42b4-a3bb-df8621dd58cb\") " pod="openshift-dns/dns-default-c6gvb"
Sep 18 13:59:44.128123 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.128098    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ccb6a3bc-51ff-46d8-8a67-a97bd76f618c-bound-sa-token\") pod \"image-registry-549d599756-4kgjb\" (UID: \"ccb6a3bc-51ff-46d8-8a67-a97bd76f618c\") " pod="openshift-image-registry/image-registry-549d599756-4kgjb"
Sep 18 13:59:44.129304 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.129285    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vrld\" (UniqueName: \"kubernetes.io/projected/ccb6a3bc-51ff-46d8-8a67-a97bd76f618c-kube-api-access-8vrld\") pod \"image-registry-549d599756-4kgjb\" (UID: \"ccb6a3bc-51ff-46d8-8a67-a97bd76f618c\") " pod="openshift-image-registry/image-registry-549d599756-4kgjb"
Sep 18 13:59:44.130145 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.130127    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr9tb\" (UniqueName: \"kubernetes.io/projected/7903d08d-219b-453f-9a68-3710e004f0e4-kube-api-access-zr9tb\") pod \"ingress-canary-gjcx2\" (UID: \"7903d08d-219b-453f-9a68-3710e004f0e4\") " pod="openshift-ingress-canary/ingress-canary-gjcx2"
Sep 18 13:59:44.139416 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.139378    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:44.168609 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.167500    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c6gvb"
Sep 18 13:59:44.172034 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.171972    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-549d599756-4kgjb"
Sep 18 13:59:44.190479 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.190395    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gjcx2"
Sep 18 13:59:44.201003 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.200965    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5ac8c4ef-13cb-43f0-b07b-5540874267d3-data-volume\") pod \"insights-runtime-extractor-52cv4\" (UID: \"5ac8c4ef-13cb-43f0-b07b-5540874267d3\") " pod="openshift-insights/insights-runtime-extractor-52cv4"
Sep 18 13:59:44.201099 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.201020    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5ac8c4ef-13cb-43f0-b07b-5540874267d3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-52cv4\" (UID: \"5ac8c4ef-13cb-43f0-b07b-5540874267d3\") " pod="openshift-insights/insights-runtime-extractor-52cv4"
Sep 18 13:59:44.201139 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.201101    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t55zn\" (UniqueName: \"kubernetes.io/projected/5ac8c4ef-13cb-43f0-b07b-5540874267d3-kube-api-access-t55zn\") pod \"insights-runtime-extractor-52cv4\" (UID: \"5ac8c4ef-13cb-43f0-b07b-5540874267d3\") " pod="openshift-insights/insights-runtime-extractor-52cv4"
Sep 18 13:59:44.201202 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.201149    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5ac8c4ef-13cb-43f0-b07b-5540874267d3-crio-socket\") pod \"insights-runtime-extractor-52cv4\" (UID: \"5ac8c4ef-13cb-43f0-b07b-5540874267d3\") " pod="openshift-insights/insights-runtime-extractor-52cv4"
Sep 18 13:59:44.201291 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.201275    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5ac8c4ef-13cb-43f0-b07b-5540874267d3-crio-socket\") pod \"insights-runtime-extractor-52cv4\" (UID: \"5ac8c4ef-13cb-43f0-b07b-5540874267d3\") " pod="openshift-insights/insights-runtime-extractor-52cv4"
Sep 18 13:59:44.201398 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.201370    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5ac8c4ef-13cb-43f0-b07b-5540874267d3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-52cv4\" (UID: \"5ac8c4ef-13cb-43f0-b07b-5540874267d3\") " pod="openshift-insights/insights-runtime-extractor-52cv4"
Sep 18 13:59:44.201513 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.201490    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5ac8c4ef-13cb-43f0-b07b-5540874267d3-data-volume\") pod \"insights-runtime-extractor-52cv4\" (UID: \"5ac8c4ef-13cb-43f0-b07b-5540874267d3\") " pod="openshift-insights/insights-runtime-extractor-52cv4"
Sep 18 13:59:44.202283 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.202260    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5ac8c4ef-13cb-43f0-b07b-5540874267d3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-52cv4\" (UID: \"5ac8c4ef-13cb-43f0-b07b-5540874267d3\") " pod="openshift-insights/insights-runtime-extractor-52cv4"
Sep 18 13:59:44.203701 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.203672    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5ac8c4ef-13cb-43f0-b07b-5540874267d3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-52cv4\" (UID: \"5ac8c4ef-13cb-43f0-b07b-5540874267d3\") " pod="openshift-insights/insights-runtime-extractor-52cv4"
Sep 18 13:59:44.224758 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.224564    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t55zn\" (UniqueName: \"kubernetes.io/projected/5ac8c4ef-13cb-43f0-b07b-5540874267d3-kube-api-access-t55zn\") pod \"insights-runtime-extractor-52cv4\" (UID: \"5ac8c4ef-13cb-43f0-b07b-5540874267d3\") " pod="openshift-insights/insights-runtime-extractor-52cv4"
Sep 18 13:59:44.245253 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.245182    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9b5c4599-q9jk2"
Sep 18 13:59:44.258085 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.257992    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:44.272980 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.272949    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9b5c4599-q9jk2"
Sep 18 13:59:44.273265 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.273215    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-52cv4"
Sep 18 13:59:44.402975 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.402944    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2f629f41-2508-48d0-800b-d541840d2082-image-registry-private-configuration\") pod \"2f629f41-2508-48d0-800b-d541840d2082\" (UID: \"2f629f41-2508-48d0-800b-d541840d2082\") "
Sep 18 13:59:44.403113 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.402993    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2f629f41-2508-48d0-800b-d541840d2082-registry-tls\") pod \"2f629f41-2508-48d0-800b-d541840d2082\" (UID: \"2f629f41-2508-48d0-800b-d541840d2082\") "
Sep 18 13:59:44.403113 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.403030    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2f629f41-2508-48d0-800b-d541840d2082-ca-trust-extracted\") pod \"2f629f41-2508-48d0-800b-d541840d2082\" (UID: \"2f629f41-2508-48d0-800b-d541840d2082\") "
Sep 18 13:59:44.403113 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.403065    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2f629f41-2508-48d0-800b-d541840d2082-installation-pull-secrets\") pod \"2f629f41-2508-48d0-800b-d541840d2082\" (UID: \"2f629f41-2508-48d0-800b-d541840d2082\") "
Sep 18 13:59:44.403113 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.403095    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lznxw\" (UniqueName: \"kubernetes.io/projected/2f629f41-2508-48d0-800b-d541840d2082-kube-api-access-lznxw\") pod \"2f629f41-2508-48d0-800b-d541840d2082\" (UID: \"2f629f41-2508-48d0-800b-d541840d2082\") "
Sep 18 13:59:44.403339 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.403137    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2f629f41-2508-48d0-800b-d541840d2082-registry-certificates\") pod \"2f629f41-2508-48d0-800b-d541840d2082\" (UID: \"2f629f41-2508-48d0-800b-d541840d2082\") "
Sep 18 13:59:44.403339 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.403282    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2f629f41-2508-48d0-800b-d541840d2082-trusted-ca\") pod \"2f629f41-2508-48d0-800b-d541840d2082\" (UID: \"2f629f41-2508-48d0-800b-d541840d2082\") "
Sep 18 13:59:44.403339 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.403311    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2f629f41-2508-48d0-800b-d541840d2082-bound-sa-token\") pod \"2f629f41-2508-48d0-800b-d541840d2082\" (UID: \"2f629f41-2508-48d0-800b-d541840d2082\") "
Sep 18 13:59:44.404615 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.404579    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f629f41-2508-48d0-800b-d541840d2082-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "2f629f41-2508-48d0-800b-d541840d2082" (UID: "2f629f41-2508-48d0-800b-d541840d2082"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:59:44.404740 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.404703    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f629f41-2508-48d0-800b-d541840d2082-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "2f629f41-2508-48d0-800b-d541840d2082" (UID: "2f629f41-2508-48d0-800b-d541840d2082"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:59:44.405015 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.404974    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f629f41-2508-48d0-800b-d541840d2082-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "2f629f41-2508-48d0-800b-d541840d2082" (UID: "2f629f41-2508-48d0-800b-d541840d2082"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:59:44.405611 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.405592    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ddf856f5c-4wwwq"]
Sep 18 13:59:44.406058 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.406021    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f629f41-2508-48d0-800b-d541840d2082-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "2f629f41-2508-48d0-800b-d541840d2082" (UID: "2f629f41-2508-48d0-800b-d541840d2082"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:59:44.406531 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.406453    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f629f41-2508-48d0-800b-d541840d2082-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "2f629f41-2508-48d0-800b-d541840d2082" (UID: "2f629f41-2508-48d0-800b-d541840d2082"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:59:44.406600 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.406545    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f629f41-2508-48d0-800b-d541840d2082-kube-api-access-lznxw" (OuterVolumeSpecName: "kube-api-access-lznxw") pod "2f629f41-2508-48d0-800b-d541840d2082" (UID: "2f629f41-2508-48d0-800b-d541840d2082"). InnerVolumeSpecName "kube-api-access-lznxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:59:44.406720 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.406706    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f629f41-2508-48d0-800b-d541840d2082-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "2f629f41-2508-48d0-800b-d541840d2082" (UID: "2f629f41-2508-48d0-800b-d541840d2082"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:59:44.406814 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.406799    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f629f41-2508-48d0-800b-d541840d2082-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "2f629f41-2508-48d0-800b-d541840d2082" (UID: "2f629f41-2508-48d0-800b-d541840d2082"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:59:44.409086 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:59:44.409068    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfceb828f_1889_4dc2_8201_1e7a74d628bb.slice/crio-5122e997f6c40349b849e74622cef83b0302c4fdc2cbb2df6786119b3972b4b6 WatchSource:0}: Error finding container 5122e997f6c40349b849e74622cef83b0302c4fdc2cbb2df6786119b3972b4b6: Status 404 returned error can't find the container with id 5122e997f6c40349b849e74622cef83b0302c4fdc2cbb2df6786119b3972b4b6
Sep 18 13:59:44.434781 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.434757    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74fcd9d577-4w4hq"]
Sep 18 13:59:44.437187 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:59:44.437154    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfbe49c7e_f072_46a0_a0c1_47344a767afc.slice/crio-fc93834abbe5106d0ac588592e7b48672f88707c71b37aaf140a63992ca1726a WatchSource:0}: Error finding container fc93834abbe5106d0ac588592e7b48672f88707c71b37aaf140a63992ca1726a: Status 404 returned error can't find the container with id fc93834abbe5106d0ac588592e7b48672f88707c71b37aaf140a63992ca1726a
Sep 18 13:59:44.457731 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.457714    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d8656f554-r9bff"]
Sep 18 13:59:44.459612 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:59:44.459593    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda8ad0db_9a94_49d8_b301_42e2514bbb46.slice/crio-d9423b2c82a572c4734de706f8b8e9598aee4b2f78c3b95cb3d0690c0c894d49 WatchSource:0}: Error finding container d9423b2c82a572c4734de706f8b8e9598aee4b2f78c3b95cb3d0690c0c894d49: Status 404 returned error can't find the container with id d9423b2c82a572c4734de706f8b8e9598aee4b2f78c3b95cb3d0690c0c894d49
Sep 18 13:59:44.504306 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.504259    2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2f629f41-2508-48d0-800b-d541840d2082-bound-sa-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 13:59:44.504306 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.504280    2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2f629f41-2508-48d0-800b-d541840d2082-image-registry-private-configuration\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 13:59:44.504306 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.504293    2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2f629f41-2508-48d0-800b-d541840d2082-registry-tls\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 13:59:44.504735 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.504307    2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2f629f41-2508-48d0-800b-d541840d2082-ca-trust-extracted\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 13:59:44.504735 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.504319    2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2f629f41-2508-48d0-800b-d541840d2082-installation-pull-secrets\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 13:59:44.504735 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.504332    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lznxw\" (UniqueName: \"kubernetes.io/projected/2f629f41-2508-48d0-800b-d541840d2082-kube-api-access-lznxw\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 13:59:44.504735 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.504344    2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2f629f41-2508-48d0-800b-d541840d2082-registry-certificates\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 13:59:44.504735 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.504358    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2f629f41-2508-48d0-800b-d541840d2082-trusted-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 13:59:44.507130 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.507114    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-549d599756-4kgjb"]
Sep 18 13:59:44.508118 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:59:44.508024    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bedf54a_fdc9_42b4_a3bb_df8621dd58cb.slice/crio-f6a12029ccd6608bed366c815943e54f07349f6d7c7b2579dd41dd804e380d2d WatchSource:0}: Error finding container f6a12029ccd6608bed366c815943e54f07349f6d7c7b2579dd41dd804e380d2d: Status 404 returned error can't find the container with id f6a12029ccd6608bed366c815943e54f07349f6d7c7b2579dd41dd804e380d2d
Sep 18 13:59:44.508973 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:59:44.508948    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7903d08d_219b_453f_9a68_3710e004f0e4.slice/crio-6adb8f72a3f72d20676ab0467f1e7c8364d5b75373a302d64b2bf0b6f87ca294 WatchSource:0}: Error finding container 6adb8f72a3f72d20676ab0467f1e7c8364d5b75373a302d64b2bf0b6f87ca294: Status 404 returned error can't find the container with id 6adb8f72a3f72d20676ab0467f1e7c8364d5b75373a302d64b2bf0b6f87ca294
Sep 18 13:59:44.509546 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.509519    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gjcx2"]
Sep 18 13:59:44.509641 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:59:44.509564    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccb6a3bc_51ff_46d8_8a67_a97bd76f618c.slice/crio-420f3d771cb0858f9f5b5e4b6a7614dfacbe46bd0ccc50cf3fc18c1f1957e9ae WatchSource:0}: Error finding container 420f3d771cb0858f9f5b5e4b6a7614dfacbe46bd0ccc50cf3fc18c1f1957e9ae: Status 404 returned error can't find the container with id 420f3d771cb0858f9f5b5e4b6a7614dfacbe46bd0ccc50cf3fc18c1f1957e9ae
Sep 18 13:59:44.510311 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.510286    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c6gvb"]
Sep 18 13:59:44.568901 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.568881    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-52cv4"]
Sep 18 13:59:44.570892 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:59:44.570865    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ac8c4ef_13cb_43f0_b07b_5540874267d3.slice/crio-7a9394f5b3361c43fa38625cee99434c97999d20008e5838ed849cd4bb560bac WatchSource:0}: Error finding container 7a9394f5b3361c43fa38625cee99434c97999d20008e5838ed849cd4bb560bac: Status 404 returned error can't find the container with id 7a9394f5b3361c43fa38625cee99434c97999d20008e5838ed849cd4bb560bac
Sep 18 13:59:44.588563 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.588543    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:44.695122 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:44.695101    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:45.251803 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:45.251368    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c6gvb" event={"ID":"5bedf54a-fdc9-42b4-a3bb-df8621dd58cb","Type":"ContainerStarted","Data":"f6a12029ccd6608bed366c815943e54f07349f6d7c7b2579dd41dd804e380d2d"}
Sep 18 13:59:45.253871 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:45.253809    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gjcx2" event={"ID":"7903d08d-219b-453f-9a68-3710e004f0e4","Type":"ContainerStarted","Data":"6adb8f72a3f72d20676ab0467f1e7c8364d5b75373a302d64b2bf0b6f87ca294"}
Sep 18 13:59:45.256929 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:45.256103    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-549d599756-4kgjb" event={"ID":"ccb6a3bc-51ff-46d8-8a67-a97bd76f618c","Type":"ContainerStarted","Data":"661ea6415cc229b48b6b38cc052cd2176f2af1a41fea18b95ce7ec6f7debf6d0"}
Sep 18 13:59:45.256929 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:45.256150    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-549d599756-4kgjb" event={"ID":"ccb6a3bc-51ff-46d8-8a67-a97bd76f618c","Type":"ContainerStarted","Data":"420f3d771cb0858f9f5b5e4b6a7614dfacbe46bd0ccc50cf3fc18c1f1957e9ae"}
Sep 18 13:59:45.256929 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:45.256878    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-549d599756-4kgjb"
Sep 18 13:59:45.259539 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:45.259487    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d8656f554-r9bff" event={"ID":"da8ad0db-9a94-49d8-b301-42e2514bbb46","Type":"ContainerStarted","Data":"d9423b2c82a572c4734de706f8b8e9598aee4b2f78c3b95cb3d0690c0c894d49"}
Sep 18 13:59:45.263992 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:45.263694    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74fcd9d577-4w4hq" event={"ID":"fbe49c7e-f072-46a0-a0c1-47344a767afc","Type":"ContainerStarted","Data":"fc93834abbe5106d0ac588592e7b48672f88707c71b37aaf140a63992ca1726a"}
Sep 18 13:59:45.267825 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:45.267800    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-52cv4" event={"ID":"5ac8c4ef-13cb-43f0-b07b-5540874267d3","Type":"ContainerStarted","Data":"bb4d6baa6d3636ee421c3f4459905c561bbe58f9133815e506b3e357c21b87ec"}
Sep 18 13:59:45.267931 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:45.267856    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-52cv4" event={"ID":"5ac8c4ef-13cb-43f0-b07b-5540874267d3","Type":"ContainerStarted","Data":"7a9394f5b3361c43fa38625cee99434c97999d20008e5838ed849cd4bb560bac"}
Sep 18 13:59:45.272606 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:45.272548    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ddf856f5c-4wwwq" event={"ID":"fceb828f-1889-4dc2-8201-1e7a74d628bb","Type":"ContainerStarted","Data":"5122e997f6c40349b849e74622cef83b0302c4fdc2cbb2df6786119b3972b4b6"}
Sep 18 13:59:45.272606 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:45.272593    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9b5c4599-q9jk2"
Sep 18 13:59:45.311061 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:45.310856    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-549d599756-4kgjb" podStartSLOduration=2.310835971 podStartE2EDuration="2.310835971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:59:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:59:45.282467791 +0000 UTC m=+57.171183723" watchObservedRunningTime="2026-09-18 13:59:45.310835971 +0000 UTC m=+57.199551904"
Sep 18 13:59:45.325518 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:45.325473    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-9b5c4599-q9jk2"]
Sep 18 13:59:45.330100 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:45.329998    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-9b5c4599-q9jk2"]
Sep 18 13:59:46.037330 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:46.037292    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f629f41-2508-48d0-800b-d541840d2082" path="/var/lib/kubelet/pods/2f629f41-2508-48d0-800b-d541840d2082/volumes"
Sep 18 13:59:46.286708 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:46.286631    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-52cv4" event={"ID":"5ac8c4ef-13cb-43f0-b07b-5540874267d3","Type":"ContainerStarted","Data":"efb840e6f06d3afe72bed4223a088d98ba3314e2b7c4920d7b00e9f008a2dd70"}
Sep 18 13:59:46.863499 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:46.863307    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:46.968346 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:46.968315    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:47.048216 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:47.048166    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:49.193807 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:49.193777    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sg4tv"
Sep 18 13:59:49.365779 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:49.365746    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:49.467021 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:49.466938    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:49.564892 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:49.564854    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:50.080460 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:50.080426    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:52.008250 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:52.008202    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:52.265010 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:52.264925    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:52.348506 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:52.347993    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-x4bcs"]
Sep 18 13:59:52.350327 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:52.350305    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-x4bcs"
Sep 18 13:59:52.354613 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:52.354592    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fvvcw\""
Sep 18 13:59:52.354613 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:52.354612    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 18 13:59:52.354786 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:52.354638    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 13:59:52.354786 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:52.354658    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 13:59:52.354977 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:52.354958    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 18 13:59:52.355393 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:52.355373    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 18 13:59:52.355518 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:52.355373    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 18 13:59:52.369730 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:52.369703    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d3a439e0-fb98-4dab-a637-63d2d17ae8b1-node-exporter-tls\") pod \"node-exporter-x4bcs\" (UID: \"d3a439e0-fb98-4dab-a637-63d2d17ae8b1\") " pod="openshift-monitoring/node-exporter-x4bcs"
Sep 18 13:59:52.369814 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:52.369739    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d3a439e0-fb98-4dab-a637-63d2d17ae8b1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x4bcs\" (UID: \"d3a439e0-fb98-4dab-a637-63d2d17ae8b1\") " pod="openshift-monitoring/node-exporter-x4bcs"
Sep 18 13:59:52.369854 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:52.369830    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d3a439e0-fb98-4dab-a637-63d2d17ae8b1-node-exporter-accelerators-collector-config\") pod \"node-exporter-x4bcs\" (UID: \"d3a439e0-fb98-4dab-a637-63d2d17ae8b1\") " pod="openshift-monitoring/node-exporter-x4bcs"
Sep 18 13:59:52.369888 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:52.369875    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpcvv\" (UniqueName: \"kubernetes.io/projected/d3a439e0-fb98-4dab-a637-63d2d17ae8b1-kube-api-access-mpcvv\") pod \"node-exporter-x4bcs\" (UID: \"d3a439e0-fb98-4dab-a637-63d2d17ae8b1\") " pod="openshift-monitoring/node-exporter-x4bcs"
Sep 18 13:59:52.369931 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:52.369901    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d3a439e0-fb98-4dab-a637-63d2d17ae8b1-root\") pod \"node-exporter-x4bcs\" (UID: \"d3a439e0-fb98-4dab-a637-63d2d17ae8b1\") " pod="openshift-monitoring/node-exporter-x4bcs"
Sep 18 13:59:52.369969 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:52.369935    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d3a439e0-fb98-4dab-a637-63d2d17ae8b1-node-exporter-wtmp\") pod \"node-exporter-x4bcs\" (UID: \"d3a439e0-fb98-4dab-a637-63d2d17ae8b1\") " pod="openshift-monitoring/node-exporter-x4bcs"
Sep 18 13:59:52.370013 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:52.369963    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3a439e0-fb98-4dab-a637-63d2d17ae8b1-sys\") pod \"node-exporter-x4bcs\" (UID: \"d3a439e0-fb98-4dab-a637-63d2d17ae8b1\") " pod="openshift-monitoring/node-exporter-x4bcs"
Sep 18 13:59:52.370058 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:52.370011    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d3a439e0-fb98-4dab-a637-63d2d17ae8b1-node-exporter-textfile\") pod \"node-exporter-x4bcs\" (UID: \"d3a439e0-fb98-4dab-a637-63d2d17ae8b1\") " pod="openshift-monitoring/node-exporter-x4bcs"
Sep 18 13:59:52.370058 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:52.370027    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3a439e0-fb98-4dab-a637-63d2d17ae8b1-metrics-client-ca\") pod \"node-exporter-x4bcs\" (UID: \"d3a439e0-fb98-4dab-a637-63d2d17ae8b1\") " pod="openshift-monitoring/node-exporter-x4bcs"
Sep 18 13:59:52.414226 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:52.414200    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:52.470319 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:52.470297    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d3a439e0-fb98-4dab-a637-63d2d17ae8b1-node-exporter-textfile\") pod \"node-exporter-x4bcs\" (UID: \"d3a439e0-fb98-4dab-a637-63d2d17ae8b1\") " pod="openshift-monitoring/node-exporter-x4bcs"
Sep 18 13:59:52.470422 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:52.470324    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3a439e0-fb98-4dab-a637-63d2d17ae8b1-metrics-client-ca\") pod \"node-exporter-x4bcs\" (UID: \"d3a439e0-fb98-4dab-a637-63d2d17ae8b1\") " pod="openshift-monitoring/node-exporter-x4bcs"
Sep 18 13:59:52.470422 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:52.470352    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d3a439e0-fb98-4dab-a637-63d2d17ae8b1-node-exporter-tls\") pod \"node-exporter-x4bcs\" (UID: \"d3a439e0-fb98-4dab-a637-63d2d17ae8b1\") " pod="openshift-monitoring/node-exporter-x4bcs"
Sep 18 13:59:52.470422 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:52.470375    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d3a439e0-fb98-4dab-a637-63d2d17ae8b1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x4bcs\" (UID: \"d3a439e0-fb98-4dab-a637-63d2d17ae8b1\") " pod="openshift-monitoring/node-exporter-x4bcs"
Sep 18 13:59:52.470569 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:52.470475    2579 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 18 13:59:52.470569 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:52.470524    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d3a439e0-fb98-4dab-a637-63d2d17ae8b1-node-exporter-accelerators-collector-config\") pod \"node-exporter-x4bcs\" (UID: \"d3a439e0-fb98-4dab-a637-63d2d17ae8b1\") " pod="openshift-monitoring/node-exporter-x4bcs"
Sep 18 13:59:52.470569 ip-10-0-143-120 kubenswrapper[2579]: E0918 13:59:52.470539    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3a439e0-fb98-4dab-a637-63d2d17ae8b1-node-exporter-tls podName:d3a439e0-fb98-4dab-a637-63d2d17ae8b1 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:52.970518575 +0000 UTC m=+64.859234647 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/d3a439e0-fb98-4dab-a637-63d2d17ae8b1-node-exporter-tls") pod "node-exporter-x4bcs" (UID: "d3a439e0-fb98-4dab-a637-63d2d17ae8b1") : secret "node-exporter-tls" not found
Sep 18 13:59:52.470713 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:52.470583    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpcvv\" (UniqueName: \"kubernetes.io/projected/d3a439e0-fb98-4dab-a637-63d2d17ae8b1-kube-api-access-mpcvv\") pod \"node-exporter-x4bcs\" (UID: \"d3a439e0-fb98-4dab-a637-63d2d17ae8b1\") " pod="openshift-monitoring/node-exporter-x4bcs"
Sep 18 13:59:52.470713 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:52.470612    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d3a439e0-fb98-4dab-a637-63d2d17ae8b1-root\") pod \"node-exporter-x4bcs\" (UID: \"d3a439e0-fb98-4dab-a637-63d2d17ae8b1\") " pod="openshift-monitoring/node-exporter-x4bcs"
Sep 18 13:59:52.470713 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:52.470640    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d3a439e0-fb98-4dab-a637-63d2d17ae8b1-node-exporter-wtmp\") pod \"node-exporter-x4bcs\" (UID: \"d3a439e0-fb98-4dab-a637-63d2d17ae8b1\") " pod="openshift-monitoring/node-exporter-x4bcs"
Sep 18 13:59:52.470713 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:52.470664    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3a439e0-fb98-4dab-a637-63d2d17ae8b1-sys\") pod \"node-exporter-x4bcs\" (UID: \"d3a439e0-fb98-4dab-a637-63d2d17ae8b1\") " pod="openshift-monitoring/node-exporter-x4bcs"
Sep 18 13:59:52.470889 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:52.470746    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3a439e0-fb98-4dab-a637-63d2d17ae8b1-sys\") pod \"node-exporter-x4bcs\" (UID: \"d3a439e0-fb98-4dab-a637-63d2d17ae8b1\") " pod="openshift-monitoring/node-exporter-x4bcs"
Sep 18 13:59:52.470889 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:52.470787    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d3a439e0-fb98-4dab-a637-63d2d17ae8b1-root\") pod \"node-exporter-x4bcs\" (UID: \"d3a439e0-fb98-4dab-a637-63d2d17ae8b1\") " pod="openshift-monitoring/node-exporter-x4bcs"
Sep 18 13:59:52.470889 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:52.470853    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d3a439e0-fb98-4dab-a637-63d2d17ae8b1-node-exporter-wtmp\") pod \"node-exporter-x4bcs\" (UID: \"d3a439e0-fb98-4dab-a637-63d2d17ae8b1\") " pod="openshift-monitoring/node-exporter-x4bcs"
Sep 18 13:59:52.471066 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:52.471047    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d3a439e0-fb98-4dab-a637-63d2d17ae8b1-node-exporter-textfile\") pod \"node-exporter-x4bcs\" (UID: \"d3a439e0-fb98-4dab-a637-63d2d17ae8b1\") " pod="openshift-monitoring/node-exporter-x4bcs"
Sep 18 13:59:52.471066 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:52.471059    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3a439e0-fb98-4dab-a637-63d2d17ae8b1-metrics-client-ca\") pod \"node-exporter-x4bcs\" (UID: \"d3a439e0-fb98-4dab-a637-63d2d17ae8b1\") " pod="openshift-monitoring/node-exporter-x4bcs"
Sep 18 13:59:52.471145 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:52.471053    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d3a439e0-fb98-4dab-a637-63d2d17ae8b1-node-exporter-accelerators-collector-config\") pod \"node-exporter-x4bcs\" (UID: \"d3a439e0-fb98-4dab-a637-63d2d17ae8b1\") " pod="openshift-monitoring/node-exporter-x4bcs"
Sep 18 13:59:52.472829 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:52.472807    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d3a439e0-fb98-4dab-a637-63d2d17ae8b1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x4bcs\" (UID: \"d3a439e0-fb98-4dab-a637-63d2d17ae8b1\") " pod="openshift-monitoring/node-exporter-x4bcs"
Sep 18 13:59:52.483817 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:52.483794    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpcvv\" (UniqueName: \"kubernetes.io/projected/d3a439e0-fb98-4dab-a637-63d2d17ae8b1-kube-api-access-mpcvv\") pod \"node-exporter-x4bcs\" (UID: \"d3a439e0-fb98-4dab-a637-63d2d17ae8b1\") " pod="openshift-monitoring/node-exporter-x4bcs"
Sep 18 13:59:52.675820 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:52.675792    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:52.898644 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:52.898558    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:52.974503 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:52.974473    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d3a439e0-fb98-4dab-a637-63d2d17ae8b1-node-exporter-tls\") pod \"node-exporter-x4bcs\" (UID: \"d3a439e0-fb98-4dab-a637-63d2d17ae8b1\") " pod="openshift-monitoring/node-exporter-x4bcs"
Sep 18 13:59:52.976850 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:52.976827    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d3a439e0-fb98-4dab-a637-63d2d17ae8b1-node-exporter-tls\") pod \"node-exporter-x4bcs\" (UID: \"d3a439e0-fb98-4dab-a637-63d2d17ae8b1\") " pod="openshift-monitoring/node-exporter-x4bcs"
Sep 18 13:59:53.033305 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:53.033232    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:53.261602 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:53.261569    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-x4bcs"
Sep 18 13:59:53.264725 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:53.264699    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:53.272611 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:59:53.272590    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3a439e0_fb98_4dab_a637_63d2d17ae8b1.slice/crio-35cba21c175093639cb275703c4bdbd624ecb9064f705c352b489c7c5099e782 WatchSource:0}: Error finding container 35cba21c175093639cb275703c4bdbd624ecb9064f705c352b489c7c5099e782: Status 404 returned error can't find the container with id 35cba21c175093639cb275703c4bdbd624ecb9064f705c352b489c7c5099e782
Sep 18 13:59:53.308630 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:53.308606    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gjcx2" event={"ID":"7903d08d-219b-453f-9a68-3710e004f0e4","Type":"ContainerStarted","Data":"b67093aa446b826dc8d5e6db631348943306b2f74804d8181f08c14fd74c4517"}
Sep 18 13:59:53.310098 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:53.310073    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d8656f554-r9bff" event={"ID":"da8ad0db-9a94-49d8-b301-42e2514bbb46","Type":"ContainerStarted","Data":"ef29a19d497ed6cfc2caaf268ebfe5152ad338f67815cbf5fe7ce1fdc927e4e2"}
Sep 18 13:59:53.312025 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:53.312008    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74fcd9d577-4w4hq" event={"ID":"fbe49c7e-f072-46a0-a0c1-47344a767afc","Type":"ContainerStarted","Data":"a24f1c0e899b3548d3754d7965c5bb53b75674cfd21e8a6389fab702fb54c79f"}
Sep 18 13:59:53.312094 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:53.312038    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74fcd9d577-4w4hq" event={"ID":"fbe49c7e-f072-46a0-a0c1-47344a767afc","Type":"ContainerStarted","Data":"fbf37c3760c31007c149c307f2ce7c15ee729df7273b7d61ec6acd3c74a826a8"}
Sep 18 13:59:53.312094 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:53.312056    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74fcd9d577-4w4hq" event={"ID":"fbe49c7e-f072-46a0-a0c1-47344a767afc","Type":"ContainerStarted","Data":"8a6a4b72deedcbd1e75d72702cdf5e55f0f7e4fae677747a8c5f207ee54fdc3e"}
Sep 18 13:59:53.313225 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:53.313201    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x4bcs" event={"ID":"d3a439e0-fb98-4dab-a637-63d2d17ae8b1","Type":"ContainerStarted","Data":"35cba21c175093639cb275703c4bdbd624ecb9064f705c352b489c7c5099e782"}
Sep 18 13:59:53.315152 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:53.315133    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-52cv4" event={"ID":"5ac8c4ef-13cb-43f0-b07b-5540874267d3","Type":"ContainerStarted","Data":"52fc9dc330effa1fae85debdf7aacd12b8285df7218d7e7dba1c5e167b0e2dfa"}
Sep 18 13:59:53.316576 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:53.316554    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ddf856f5c-4wwwq" event={"ID":"fceb828f-1889-4dc2-8201-1e7a74d628bb","Type":"ContainerStarted","Data":"c773801b1f602c481b624c85e268cce7e2d1bdc735f8a16ff15ecc396f7c9aa1"}
Sep 18 13:59:53.316719 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:53.316705    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ddf856f5c-4wwwq"
Sep 18 13:59:53.317947 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:53.317928    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ddf856f5c-4wwwq"
Sep 18 13:59:53.318271 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:53.318253    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c6gvb" event={"ID":"5bedf54a-fdc9-42b4-a3bb-df8621dd58cb","Type":"ContainerStarted","Data":"f30742b21ae69d6301c736a689c024239f4011b29fc960b8ccc5c778efbd6197"}
Sep 18 13:59:53.318345 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:53.318291    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c6gvb" event={"ID":"5bedf54a-fdc9-42b4-a3bb-df8621dd58cb","Type":"ContainerStarted","Data":"b72a5162c359ca567ac33e0cac84ae2921c643fa68aea7f457fb74fd2f369a7b"}
Sep 18 13:59:53.318399 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:53.318368    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-c6gvb"
Sep 18 13:59:53.335582 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:53.335544    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-gjcx2" podStartSLOduration=1.8725307789999999 podStartE2EDuration="10.335531721s" totalImagesPullingTime="8.463000942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:59:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:59:53.335239235 +0000 UTC m=+65.223955166" watchObservedRunningTime="2026-09-18 13:59:53.335531721 +0000 UTC m=+65.224247630"
Sep 18 13:59:53.362363 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:53.362197    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-c6gvb" podStartSLOduration=1.8984289859999999 podStartE2EDuration="10.362167265s" totalImagesPullingTime="8.463738279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:59:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:59:53.361714349 +0000 UTC m=+65.250430282" watchObservedRunningTime="2026-09-18 13:59:53.362167265 +0000 UTC m=+65.250883196"
Sep 18 13:59:53.381368 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:53.381332    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ddf856f5c-4wwwq" podStartSLOduration=1.791980407 podStartE2EDuration="10.381319772s" totalImagesPullingTime="8.589339365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:59:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:59:53.38099422 +0000 UTC m=+65.269710152" watchObservedRunningTime="2026-09-18 13:59:53.381319772 +0000 UTC m=+65.270035727"
Sep 18 13:59:53.387939 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:53.387917    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:53.414880 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:53.414814    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d8656f554-r9bff" podStartSLOduration=1.89873518 podStartE2EDuration="10.41480209s" totalImagesPullingTime="8.51606691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:59:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:59:53.413602452 +0000 UTC m=+65.302318383" watchObservedRunningTime="2026-09-18 13:59:53.41480209 +0000 UTC m=+65.303518015"
Sep 18 13:59:53.437876 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:53.437835    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-52cv4" podStartSLOduration=2.265198516 podStartE2EDuration="10.43782503s" totalImagesPullingTime="8.172626514s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:59:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:59:53.437257052 +0000 UTC m=+65.325972983" watchObservedRunningTime="2026-09-18 13:59:53.43782503 +0000 UTC m=+65.326540962"
Sep 18 13:59:53.464221 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:53.464181    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74fcd9d577-4w4hq" podStartSLOduration=1.902297247 podStartE2EDuration="10.46415854s" totalImagesPullingTime="8.561861293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:59:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:59:53.463241709 +0000 UTC m=+65.351957640" watchObservedRunningTime="2026-09-18 13:59:53.46415854 +0000 UTC m=+65.352874484"
Sep 18 13:59:53.527556 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:53.527532    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:54.323288 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:54.323257    2579 generic.go:356] "Generic (PLEG): container finished" podID="d3a439e0-fb98-4dab-a637-63d2d17ae8b1" containerID="6c309414ae37927e5156740b48b571f15b22d26d9b9125847095a3a8d44935ae" exitCode=0
Sep 18 13:59:54.323648 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:54.323357    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x4bcs" event={"ID":"d3a439e0-fb98-4dab-a637-63d2d17ae8b1","Type":"ContainerDied","Data":"6c309414ae37927e5156740b48b571f15b22d26d9b9125847095a3a8d44935ae"}
Sep 18 13:59:55.329373 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:55.329335    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x4bcs" event={"ID":"d3a439e0-fb98-4dab-a637-63d2d17ae8b1","Type":"ContainerStarted","Data":"138e26c619d72fd2ea8276817dc16f19099bb636ad4fa413656e4d01c7185da5"}
Sep 18 13:59:55.329741 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:55.329431    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x4bcs" event={"ID":"d3a439e0-fb98-4dab-a637-63d2d17ae8b1","Type":"ContainerStarted","Data":"22c66a4705979dfcc6c5220a3a844f22eae3385511ae5d06d0860fc0473a628b"}
Sep 18 13:59:55.353116 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:55.353067    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-x4bcs" podStartSLOduration=2.488523709 podStartE2EDuration="3.353052241s" totalImagesPullingTime="802.085165ms" totalInitContainerRuntime="62.443367ms" isStatefulPod=true podCreationTimestamp="2026-09-18 13:59:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:59:55.352818228 +0000 UTC m=+67.241534159" watchObservedRunningTime="2026-09-18 13:59:55.353052241 +0000 UTC m=+67.241768172"
Sep 18 13:59:55.440035 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:55.440005    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:55.533351 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:55.533313    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:55.633105 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:55.633047    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:55.698913 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:55.698887    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57282ff1-fab4-4cd4-8087-5517bd249f10-metrics-certs\") pod \"network-metrics-daemon-2mgfl\" (UID: \"57282ff1-fab4-4cd4-8087-5517bd249f10\") " pod="openshift-multus/network-metrics-daemon-2mgfl"
Sep 18 13:59:55.701405 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:55.701384    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 13:59:55.711656 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:55.711628    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57282ff1-fab4-4cd4-8087-5517bd249f10-metrics-certs\") pod \"network-metrics-daemon-2mgfl\" (UID: \"57282ff1-fab4-4cd4-8087-5517bd249f10\") " pod="openshift-multus/network-metrics-daemon-2mgfl"
Sep 18 13:59:55.800028 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:55.799996    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t8sk\" (UniqueName: \"kubernetes.io/projected/7e8de495-7804-4c38-af57-655459217f1c-kube-api-access-4t8sk\") pod \"network-check-target-j7msq\" (UID: \"7e8de495-7804-4c38-af57-655459217f1c\") " pod="openshift-network-diagnostics/network-check-target-j7msq"
Sep 18 13:59:55.800152 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:55.800036    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ca72836e-7ed0-47e5-a106-2f47340eb6b1-original-pull-secret\") pod \"global-pull-secret-syncer-2hvmm\" (UID: \"ca72836e-7ed0-47e5-a106-2f47340eb6b1\") " pod="kube-system/global-pull-secret-syncer-2hvmm"
Sep 18 13:59:55.802471 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:55.802448    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 13:59:55.802582 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:55.802501    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 13:59:55.812907 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:55.812879    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 13:59:55.813111 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:55.813091    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ca72836e-7ed0-47e5-a106-2f47340eb6b1-original-pull-secret\") pod \"global-pull-secret-syncer-2hvmm\" (UID: \"ca72836e-7ed0-47e5-a106-2f47340eb6b1\") " pod="kube-system/global-pull-secret-syncer-2hvmm"
Sep 18 13:59:55.823984 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:55.823963    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t8sk\" (UniqueName: \"kubernetes.io/projected/7e8de495-7804-4c38-af57-655459217f1c-kube-api-access-4t8sk\") pod \"network-check-target-j7msq\" (UID: \"7e8de495-7804-4c38-af57-655459217f1c\") " pod="openshift-network-diagnostics/network-check-target-j7msq"
Sep 18 13:59:55.853743 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:55.853717    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8cnh9\""
Sep 18 13:59:55.854794 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:55.854778    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2hvmm"
Sep 18 13:59:55.861370 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:55.861350    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2mgfl"
Sep 18 13:59:55.903197 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:55.903044    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:55.946670 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:55.946435    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-z6j5p\""
Sep 18 13:59:55.954491 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:55.954470    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j7msq"
Sep 18 13:59:56.126275 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:56.126249    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2hvmm"]
Sep 18 13:59:56.129468 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:59:56.129446    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca72836e_7ed0_47e5_a106_2f47340eb6b1.slice/crio-39a04fe0867386502085c10523bbaec19b41c7e7e119693d322019d13188c348 WatchSource:0}: Error finding container 39a04fe0867386502085c10523bbaec19b41c7e7e119693d322019d13188c348: Status 404 returned error can't find the container with id 39a04fe0867386502085c10523bbaec19b41c7e7e119693d322019d13188c348
Sep 18 13:59:56.138592 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:56.138542    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2mgfl"]
Sep 18 13:59:56.140753 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:59:56.140732    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57282ff1_fab4_4cd4_8087_5517bd249f10.slice/crio-609d6fcbae51fab391802f0b10ff5635a32e47a7bacc53d0960e1f3974f6f620 WatchSource:0}: Error finding container 609d6fcbae51fab391802f0b10ff5635a32e47a7bacc53d0960e1f3974f6f620: Status 404 returned error can't find the container with id 609d6fcbae51fab391802f0b10ff5635a32e47a7bacc53d0960e1f3974f6f620
Sep 18 13:59:56.221380 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:56.221332    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j7msq"]
Sep 18 13:59:56.223835 ip-10-0-143-120 kubenswrapper[2579]: W0918 13:59:56.223814    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e8de495_7804_4c38_af57_655459217f1c.slice/crio-b016f7b8ddfe3dbaddc11990828afc07967e75b295998a862e3174bcc81c9f8e WatchSource:0}: Error finding container b016f7b8ddfe3dbaddc11990828afc07967e75b295998a862e3174bcc81c9f8e: Status 404 returned error can't find the container with id b016f7b8ddfe3dbaddc11990828afc07967e75b295998a862e3174bcc81c9f8e
Sep 18 13:59:56.334100 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:56.334070    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2mgfl" event={"ID":"57282ff1-fab4-4cd4-8087-5517bd249f10","Type":"ContainerStarted","Data":"609d6fcbae51fab391802f0b10ff5635a32e47a7bacc53d0960e1f3974f6f620"}
Sep 18 13:59:56.335293 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:56.335269    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2hvmm" event={"ID":"ca72836e-7ed0-47e5-a106-2f47340eb6b1","Type":"ContainerStarted","Data":"39a04fe0867386502085c10523bbaec19b41c7e7e119693d322019d13188c348"}
Sep 18 13:59:56.336466 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:56.336442    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j7msq" event={"ID":"7e8de495-7804-4c38-af57-655459217f1c","Type":"ContainerStarted","Data":"b016f7b8ddfe3dbaddc11990828afc07967e75b295998a862e3174bcc81c9f8e"}
Sep 18 13:59:57.014811 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:57.014777    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:57.173932 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:57.173898    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:57.343165 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:57.343134    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2mgfl" event={"ID":"57282ff1-fab4-4cd4-8087-5517bd249f10","Type":"ContainerStarted","Data":"522077d66231e0e4b32d862bb96368ac6650e0e135efa732296b68652650ef27"}
Sep 18 13:59:58.349179 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:58.349140    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2mgfl" event={"ID":"57282ff1-fab4-4cd4-8087-5517bd249f10","Type":"ContainerStarted","Data":"c89e5a40c3b5bc3a0d37bb34e6963983c8c9b4b81ce4329ebee525dcc00a1d0e"}
Sep 18 13:59:58.596560 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:58.596514    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:58.702801 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:58.702734    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:58.824804 ip-10-0-143-120 kubenswrapper[2579]: I0918 13:59:58.824766    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:01.362821 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:00:01.362784    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2hvmm" event={"ID":"ca72836e-7ed0-47e5-a106-2f47340eb6b1","Type":"ContainerStarted","Data":"0eec54390f49d8ee59705607bc3daa40b6ca486b6b71504ccf09681a7984bd5d"}
Sep 18 14:00:01.364417 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:00:01.364397    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j7msq" event={"ID":"7e8de495-7804-4c38-af57-655459217f1c","Type":"ContainerStarted","Data":"2b184d850799f44a90731aa9dadfa2c6bd5bdaaf4dd63b735f00aee24eccd37d"}
Sep 18 14:00:01.364538 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:00:01.364525    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-j7msq"
Sep 18 14:00:01.380589 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:00:01.380551    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2hvmm" podStartSLOduration=66.588859661 podStartE2EDuration="1m11.380539998s" totalImagesPullingTime="4.791680337s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:58:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:00:01.379697783 +0000 UTC m=+73.268413713" watchObservedRunningTime="2026-09-18 14:00:01.380539998 +0000 UTC m=+73.269255955"
Sep 18 14:00:01.380682 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:00:01.380664    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-2mgfl" podStartSLOduration=70.381945123 podStartE2EDuration="1m11.380660095s" totalImagesPullingTime="998.714972ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:58:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:59:58.377010062 +0000 UTC m=+70.265725996" watchObservedRunningTime="2026-09-18 14:00:01.380660095 +0000 UTC m=+73.269376026"
Sep 18 14:00:01.396140 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:00:01.396102    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-j7msq" podStartSLOduration=66.704482602 podStartE2EDuration="1m11.396094104s" totalImagesPullingTime="4.691611502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:58:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:00:01.395774523 +0000 UTC m=+73.284490454" watchObservedRunningTime="2026-09-18 14:00:01.396094104 +0000 UTC m=+73.284810034"
Sep 18 14:00:01.869241 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:00:01.869198    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:01.926221 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:00:01.926190    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:02.724845 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:00:02.724683    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:02.978605 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:00:02.978483    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:03.080634 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:00:03.080608    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:03.326125 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:00:03.326096    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-c6gvb"
Sep 18 14:00:03.341555 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:00:03.341502    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:03.441220 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:00:03.441192    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:03.753548 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:00:03.753437    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:03.843337 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:00:03.843306    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:04.455696 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:00:04.455657    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:04.736533 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:00:04.736401    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:04.796766 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:00:04.796738    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:05.341209 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:00:05.341164    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:05.580002 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:00:05.579960    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:06.531010 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:00:06.530960    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:06.633457 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:00:06.633404    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:07.294936 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:00:07.294910    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-549d599756-4kgjb"
Sep 18 14:00:07.476307 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:00:07.476261    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:07.576328 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:00:07.576302    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:07.658376 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:00:07.658346    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:10.215908 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:00:10.215877    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:10.303247 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:00:10.303222    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:11.863051 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:00:11.863015    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:11.963470 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:00:11.963435    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:15.607156 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:00:15.607074    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:15.707854 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:00:15.707829    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:16.754250 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:00:16.754216    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:16.852717 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:00:16.852688    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:19.249809 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:00:19.249777    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:19.350386 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:00:19.350360    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:19.450643 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:00:19.450618    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:19.741698 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:00:19.741667    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:19.800786 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:00:19.800760    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:20.863905 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:00:20.863874    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:24.964152 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:00:24.964124    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:25.065245 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:00:25.065210    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:26.085164 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:00:26.085131    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:26.142546 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:00:26.142522    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:30.046307 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:00:30.046271    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:30.147399 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:00:30.147369    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:30.207258 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:00:30.207236    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:32.370987 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:00:32.370955    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-j7msq"
Sep 18 14:00:41.095750 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:00:41.095707    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:00:41.158456 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:00:41.158401    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:00.290666 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:01:00.290621    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:00.381565 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:01:00.381534    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:11.886705 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:01:11.886661    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:11.986565 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:01:11.986534    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:19.839523 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:01:19.839492    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:19.940549 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:01:19.940523    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:20.039959 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:01:20.039893    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:20.580259 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:01:20.580227    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:20.680864 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:01:20.680836    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:27.102816 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:01:27.102782    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:27.200595 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:01:27.200569    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:27.553396 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:01:27.553362    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:27.626551 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:01:27.626525    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:28.227009 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:01:28.226980    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:38.460897 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:01:38.460818    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:38.558689 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:01:38.558661    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:39.016390 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:01:39.016361    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:39.221924 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:01:39.221894    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:49.914004 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:01:49.913969    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:54.255712 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:01:54.255670    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:54.381510 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:01:54.381476    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:56.105253 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:01:56.105223    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:56.206609 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:01:56.206587    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:01:56.304329 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:01:56.304300    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:28.447656 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:02:28.447618    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:28.550512 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:02:28.550454    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:50.431845 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:02:50.431802    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:51.237377 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:02:51.237343    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:54.047307 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:02:54.047276    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:54.149355 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:02:54.149325    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:56.032157 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:02:56.032125    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:56.382430 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:02:56.382395    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:56.544394 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:02:56.544361    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:58.371487 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:02:58.371461    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:02:59.219043 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:02:59.219015    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:01.036891 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:01.036858    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:01.114541 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:01.114512    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:07.172432 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:07.172400    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:08.743422 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:08.743392    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:08.839940 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:08.839911    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:14.357824 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:14.357775    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:14.456934 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:14.456908    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:16.297381 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:16.297348    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:26.733088 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:26.733050    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:28.066432 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:28.066398    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:28.169324 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:28.169296    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:28.635403 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:28.635364    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:28.716055 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:28.716028    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:28.972038 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:28.971974    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:29.071149 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:29.071123    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:29.104604 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:29.104578    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-6h2ct"]
Sep 18 14:03:29.106714 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:29.106699    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-6h2ct"
Sep 18 14:03:29.108987 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:29.108963    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 18 14:03:29.108987 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:29.108979    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 18 14:03:29.109571 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:29.109461    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 18 14:03:29.109736 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:29.109716    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 18 14:03:29.111252 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:29.110355    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-ms7h5\""
Sep 18 14:03:29.127673 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:29.127651    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-6h2ct"]
Sep 18 14:03:29.205896 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:29.205873    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e3ab8f69-7fd2-46e0-94fb-98db7182cde6-webhook-cert\") pod \"opendatahub-operator-controller-manager-74d96fdcf9-6h2ct\" (UID: \"e3ab8f69-7fd2-46e0-94fb-98db7182cde6\") " pod="opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-6h2ct"
Sep 18 14:03:29.205992 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:29.205904    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e3ab8f69-7fd2-46e0-94fb-98db7182cde6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-74d96fdcf9-6h2ct\" (UID: \"e3ab8f69-7fd2-46e0-94fb-98db7182cde6\") " pod="opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-6h2ct"
Sep 18 14:03:29.205992 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:29.205921    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p8gs\" (UniqueName: \"kubernetes.io/projected/e3ab8f69-7fd2-46e0-94fb-98db7182cde6-kube-api-access-5p8gs\") pod \"opendatahub-operator-controller-manager-74d96fdcf9-6h2ct\" (UID: \"e3ab8f69-7fd2-46e0-94fb-98db7182cde6\") " pod="opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-6h2ct"
Sep 18 14:03:29.205992 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:29.205937    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e3ab8f69-7fd2-46e0-94fb-98db7182cde6-manifests\") pod \"opendatahub-operator-controller-manager-74d96fdcf9-6h2ct\" (UID: \"e3ab8f69-7fd2-46e0-94fb-98db7182cde6\") " pod="opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-6h2ct"
Sep 18 14:03:29.206102 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:29.205999    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3ab8f69-7fd2-46e0-94fb-98db7182cde6-tmp\") pod \"opendatahub-operator-controller-manager-74d96fdcf9-6h2ct\" (UID: \"e3ab8f69-7fd2-46e0-94fb-98db7182cde6\") " pod="opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-6h2ct"
Sep 18 14:03:29.306244 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:29.306225    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3ab8f69-7fd2-46e0-94fb-98db7182cde6-tmp\") pod \"opendatahub-operator-controller-manager-74d96fdcf9-6h2ct\" (UID: \"e3ab8f69-7fd2-46e0-94fb-98db7182cde6\") " pod="opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-6h2ct"
Sep 18 14:03:29.306310 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:29.306262    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e3ab8f69-7fd2-46e0-94fb-98db7182cde6-webhook-cert\") pod \"opendatahub-operator-controller-manager-74d96fdcf9-6h2ct\" (UID: \"e3ab8f69-7fd2-46e0-94fb-98db7182cde6\") " pod="opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-6h2ct"
Sep 18 14:03:29.306310 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:29.306285    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e3ab8f69-7fd2-46e0-94fb-98db7182cde6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-74d96fdcf9-6h2ct\" (UID: \"e3ab8f69-7fd2-46e0-94fb-98db7182cde6\") " pod="opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-6h2ct"
Sep 18 14:03:29.306310 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:29.306302    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p8gs\" (UniqueName: \"kubernetes.io/projected/e3ab8f69-7fd2-46e0-94fb-98db7182cde6-kube-api-access-5p8gs\") pod \"opendatahub-operator-controller-manager-74d96fdcf9-6h2ct\" (UID: \"e3ab8f69-7fd2-46e0-94fb-98db7182cde6\") " pod="opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-6h2ct"
Sep 18 14:03:29.306460 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:29.306319    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e3ab8f69-7fd2-46e0-94fb-98db7182cde6-manifests\") pod \"opendatahub-operator-controller-manager-74d96fdcf9-6h2ct\" (UID: \"e3ab8f69-7fd2-46e0-94fb-98db7182cde6\") " pod="opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-6h2ct"
Sep 18 14:03:29.306792 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:29.306766    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e3ab8f69-7fd2-46e0-94fb-98db7182cde6-manifests\") pod \"opendatahub-operator-controller-manager-74d96fdcf9-6h2ct\" (UID: \"e3ab8f69-7fd2-46e0-94fb-98db7182cde6\") " pod="opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-6h2ct"
Sep 18 14:03:29.308763 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:29.308744    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3ab8f69-7fd2-46e0-94fb-98db7182cde6-tmp\") pod \"opendatahub-operator-controller-manager-74d96fdcf9-6h2ct\" (UID: \"e3ab8f69-7fd2-46e0-94fb-98db7182cde6\") " pod="opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-6h2ct"
Sep 18 14:03:29.308961 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:29.308941    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e3ab8f69-7fd2-46e0-94fb-98db7182cde6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-74d96fdcf9-6h2ct\" (UID: \"e3ab8f69-7fd2-46e0-94fb-98db7182cde6\") " pod="opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-6h2ct"
Sep 18 14:03:29.309003 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:29.308963    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e3ab8f69-7fd2-46e0-94fb-98db7182cde6-webhook-cert\") pod \"opendatahub-operator-controller-manager-74d96fdcf9-6h2ct\" (UID: \"e3ab8f69-7fd2-46e0-94fb-98db7182cde6\") " pod="opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-6h2ct"
Sep 18 14:03:29.312957 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:29.312937    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:29.322600 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:29.322578    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p8gs\" (UniqueName: \"kubernetes.io/projected/e3ab8f69-7fd2-46e0-94fb-98db7182cde6-kube-api-access-5p8gs\") pod \"opendatahub-operator-controller-manager-74d96fdcf9-6h2ct\" (UID: \"e3ab8f69-7fd2-46e0-94fb-98db7182cde6\") " pod="opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-6h2ct"
Sep 18 14:03:29.406320 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:29.406295    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:29.420303 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:29.420282    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-6h2ct"
Sep 18 14:03:29.511227 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:29.511203    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:29.611382 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:29.611324    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:29.694797 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:29.694772    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-6h2ct"]
Sep 18 14:03:29.697482 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:29.697463    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:29.698043 ip-10-0-143-120 kubenswrapper[2579]: W0918 14:03:29.698022    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3ab8f69_7fd2_46e0_94fb_98db7182cde6.slice/crio-6a8ffeb03a796f02088144ec60c8877aea697aa7ab03174e6e6f6172cefa39a8 WatchSource:0}: Error finding container 6a8ffeb03a796f02088144ec60c8877aea697aa7ab03174e6e6f6172cefa39a8: Status 404 returned error can't find the container with id 6a8ffeb03a796f02088144ec60c8877aea697aa7ab03174e6e6f6172cefa39a8
Sep 18 14:03:30.089754 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:30.089734    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-6h2ct" event={"ID":"e3ab8f69-7fd2-46e0-94fb-98db7182cde6","Type":"ContainerStarted","Data":"6a8ffeb03a796f02088144ec60c8877aea697aa7ab03174e6e6f6172cefa39a8"}
Sep 18 14:03:33.101850 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:33.101820    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-6h2ct" event={"ID":"e3ab8f69-7fd2-46e0-94fb-98db7182cde6","Type":"ContainerStarted","Data":"06d579662797799cdc419dfaf8c7d86fb555adf228895991e30a98f051eea26a"}
Sep 18 14:03:34.108015 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:34.107977    2579 generic.go:356] "Generic (PLEG): container finished" podID="e3ab8f69-7fd2-46e0-94fb-98db7182cde6" containerID="06d579662797799cdc419dfaf8c7d86fb555adf228895991e30a98f051eea26a" exitCode=0
Sep 18 14:03:34.108404 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:34.108041    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-6h2ct" event={"ID":"e3ab8f69-7fd2-46e0-94fb-98db7182cde6","Type":"ContainerDied","Data":"06d579662797799cdc419dfaf8c7d86fb555adf228895991e30a98f051eea26a"}
Sep 18 14:03:35.113341 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:35.113305    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-6h2ct" event={"ID":"e3ab8f69-7fd2-46e0-94fb-98db7182cde6","Type":"ContainerStarted","Data":"f5beb52762ea68c81e3f94c911847466f21d1736e9c4744d9576174377991ad7"}
Sep 18 14:03:35.113738 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:35.113404    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-6h2ct"
Sep 18 14:03:35.147446 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:35.147397    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-6h2ct" podStartSLOduration=1.850335221 podStartE2EDuration="6.14737748s" totalImagesPullingTime="3.231745873s" totalInitContainerRuntime="1.065296386s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:03:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:03:35.145106157 +0000 UTC m=+287.033822088" watchObservedRunningTime="2026-09-18 14:03:35.14737748 +0000 UTC m=+287.036093468"
Sep 18 14:03:46.118795 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:46.118764    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-6h2ct"
Sep 18 14:03:47.857554 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:47.857517    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:48.344739 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:48.344705    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-86dc75d4b5-ls4xg"]
Sep 18 14:03:48.354604 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:48.354577    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-86dc75d4b5-ls4xg"
Sep 18 14:03:48.358784 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:48.358762    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 18 14:03:48.358784 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:48.358776    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 18 14:03:48.358941 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:48.358781    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-h4m78\""
Sep 18 14:03:48.359088 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:48.359074    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 18 14:03:48.362515 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:48.362498    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 18 14:03:48.367801 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:48.367780    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-86dc75d4b5-ls4xg"]
Sep 18 14:03:48.425143 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:48.425120    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2drg5\" (UniqueName: \"kubernetes.io/projected/6846cc5d-9e5f-4a5a-abbb-54c7163c1f96-kube-api-access-2drg5\") pod \"kube-auth-proxy-86dc75d4b5-ls4xg\" (UID: \"6846cc5d-9e5f-4a5a-abbb-54c7163c1f96\") " pod="openshift-ingress/kube-auth-proxy-86dc75d4b5-ls4xg"
Sep 18 14:03:48.425275 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:48.425148    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6846cc5d-9e5f-4a5a-abbb-54c7163c1f96-tmp\") pod \"kube-auth-proxy-86dc75d4b5-ls4xg\" (UID: \"6846cc5d-9e5f-4a5a-abbb-54c7163c1f96\") " pod="openshift-ingress/kube-auth-proxy-86dc75d4b5-ls4xg"
Sep 18 14:03:48.425275 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:48.425207    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6846cc5d-9e5f-4a5a-abbb-54c7163c1f96-tls-certs\") pod \"kube-auth-proxy-86dc75d4b5-ls4xg\" (UID: \"6846cc5d-9e5f-4a5a-abbb-54c7163c1f96\") " pod="openshift-ingress/kube-auth-proxy-86dc75d4b5-ls4xg"
Sep 18 14:03:48.526408 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:48.526387    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2drg5\" (UniqueName: \"kubernetes.io/projected/6846cc5d-9e5f-4a5a-abbb-54c7163c1f96-kube-api-access-2drg5\") pod \"kube-auth-proxy-86dc75d4b5-ls4xg\" (UID: \"6846cc5d-9e5f-4a5a-abbb-54c7163c1f96\") " pod="openshift-ingress/kube-auth-proxy-86dc75d4b5-ls4xg"
Sep 18 14:03:48.526515 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:48.526420    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6846cc5d-9e5f-4a5a-abbb-54c7163c1f96-tmp\") pod \"kube-auth-proxy-86dc75d4b5-ls4xg\" (UID: \"6846cc5d-9e5f-4a5a-abbb-54c7163c1f96\") " pod="openshift-ingress/kube-auth-proxy-86dc75d4b5-ls4xg"
Sep 18 14:03:48.526515 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:48.526463    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6846cc5d-9e5f-4a5a-abbb-54c7163c1f96-tls-certs\") pod \"kube-auth-proxy-86dc75d4b5-ls4xg\" (UID: \"6846cc5d-9e5f-4a5a-abbb-54c7163c1f96\") " pod="openshift-ingress/kube-auth-proxy-86dc75d4b5-ls4xg"
Sep 18 14:03:48.526621 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:03:48.526596    2579 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 18 14:03:48.526675 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:03:48.526666    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6846cc5d-9e5f-4a5a-abbb-54c7163c1f96-tls-certs podName:6846cc5d-9e5f-4a5a-abbb-54c7163c1f96 nodeName:}" failed. No retries permitted until 2026-09-18 14:03:49.026645694 +0000 UTC m=+300.915361618 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6846cc5d-9e5f-4a5a-abbb-54c7163c1f96-tls-certs") pod "kube-auth-proxy-86dc75d4b5-ls4xg" (UID: "6846cc5d-9e5f-4a5a-abbb-54c7163c1f96") : secret "kube-auth-proxy-tls" not found
Sep 18 14:03:48.528543 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:48.528525    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6846cc5d-9e5f-4a5a-abbb-54c7163c1f96-tmp\") pod \"kube-auth-proxy-86dc75d4b5-ls4xg\" (UID: \"6846cc5d-9e5f-4a5a-abbb-54c7163c1f96\") " pod="openshift-ingress/kube-auth-proxy-86dc75d4b5-ls4xg"
Sep 18 14:03:48.543309 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:48.543286    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2drg5\" (UniqueName: \"kubernetes.io/projected/6846cc5d-9e5f-4a5a-abbb-54c7163c1f96-kube-api-access-2drg5\") pod \"kube-auth-proxy-86dc75d4b5-ls4xg\" (UID: \"6846cc5d-9e5f-4a5a-abbb-54c7163c1f96\") " pod="openshift-ingress/kube-auth-proxy-86dc75d4b5-ls4xg"
Sep 18 14:03:49.029188 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:49.029135    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6846cc5d-9e5f-4a5a-abbb-54c7163c1f96-tls-certs\") pod \"kube-auth-proxy-86dc75d4b5-ls4xg\" (UID: \"6846cc5d-9e5f-4a5a-abbb-54c7163c1f96\") " pod="openshift-ingress/kube-auth-proxy-86dc75d4b5-ls4xg"
Sep 18 14:03:49.031386 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:49.031365    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6846cc5d-9e5f-4a5a-abbb-54c7163c1f96-tls-certs\") pod \"kube-auth-proxy-86dc75d4b5-ls4xg\" (UID: \"6846cc5d-9e5f-4a5a-abbb-54c7163c1f96\") " pod="openshift-ingress/kube-auth-proxy-86dc75d4b5-ls4xg"
Sep 18 14:03:49.037111 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:49.037086    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:49.263934 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:49.263910    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-86dc75d4b5-ls4xg"
Sep 18 14:03:49.545159 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:49.545114    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-86dc75d4b5-ls4xg"]
Sep 18 14:03:49.547923 ip-10-0-143-120 kubenswrapper[2579]: W0918 14:03:49.547896    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6846cc5d_9e5f_4a5a_abbb_54c7163c1f96.slice/crio-26c16ff527941879c425438a588b5e0cebebe89a6d0d5dd8773fe0f73fc52831 WatchSource:0}: Error finding container 26c16ff527941879c425438a588b5e0cebebe89a6d0d5dd8773fe0f73fc52831: Status 404 returned error can't find the container with id 26c16ff527941879c425438a588b5e0cebebe89a6d0d5dd8773fe0f73fc52831
Sep 18 14:03:49.867119 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:49.867093    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:49.875956 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:49.875932    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6kbxk"]
Sep 18 14:03:49.878444 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:49.878428    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6kbxk"
Sep 18 14:03:49.881883 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:49.881861    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 18 14:03:49.881883 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:49.881891    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 18 14:03:49.882148 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:49.881932    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 18 14:03:49.882148 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:49.881933    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 18 14:03:49.882411 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:49.882395    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-hpqqd\""
Sep 18 14:03:49.898706 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:49.898681    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6kbxk"]
Sep 18 14:03:49.930869 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:49.930845    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sg4tv_4626a667-4cd6-468c-8ef3-6e3a925c2608/ovn-acl-logging/0.log"
Sep 18 14:03:49.932127 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:49.932110    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sg4tv_4626a667-4cd6-468c-8ef3-6e3a925c2608/ovn-acl-logging/0.log"
Sep 18 14:03:49.933521 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:49.933505    2579 kubelet.go:1648] "Image garbage collection succeeded"
Sep 18 14:03:49.936228 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:49.936209    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a783cfd9-8be0-4c36-9584-2fb86d0db268-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-6kbxk\" (UID: \"a783cfd9-8be0-4c36-9584-2fb86d0db268\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6kbxk"
Sep 18 14:03:49.936312 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:49.936243    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5p9p\" (UniqueName: \"kubernetes.io/projected/a783cfd9-8be0-4c36-9584-2fb86d0db268-kube-api-access-z5p9p\") pod \"istiod-openshift-gateway-7b7995d5cd-6kbxk\" (UID: \"a783cfd9-8be0-4c36-9584-2fb86d0db268\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6kbxk"
Sep 18 14:03:49.936366 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:49.936312    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a783cfd9-8be0-4c36-9584-2fb86d0db268-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-6kbxk\" (UID: \"a783cfd9-8be0-4c36-9584-2fb86d0db268\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6kbxk"
Sep 18 14:03:49.936366 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:49.936357    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a783cfd9-8be0-4c36-9584-2fb86d0db268-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-6kbxk\" (UID: \"a783cfd9-8be0-4c36-9584-2fb86d0db268\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6kbxk"
Sep 18 14:03:49.936445 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:49.936391    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a783cfd9-8be0-4c36-9584-2fb86d0db268-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-6kbxk\" (UID: \"a783cfd9-8be0-4c36-9584-2fb86d0db268\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6kbxk"
Sep 18 14:03:49.936445 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:49.936407    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a783cfd9-8be0-4c36-9584-2fb86d0db268-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-6kbxk\" (UID: \"a783cfd9-8be0-4c36-9584-2fb86d0db268\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6kbxk"
Sep 18 14:03:49.936445 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:49.936421    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a783cfd9-8be0-4c36-9584-2fb86d0db268-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-6kbxk\" (UID: \"a783cfd9-8be0-4c36-9584-2fb86d0db268\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6kbxk"
Sep 18 14:03:49.944413 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:49.944392    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:50.036991 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:50.036875    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a783cfd9-8be0-4c36-9584-2fb86d0db268-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-6kbxk\" (UID: \"a783cfd9-8be0-4c36-9584-2fb86d0db268\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6kbxk"
Sep 18 14:03:50.036991 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:50.036912    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a783cfd9-8be0-4c36-9584-2fb86d0db268-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-6kbxk\" (UID: \"a783cfd9-8be0-4c36-9584-2fb86d0db268\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6kbxk"
Sep 18 14:03:50.036991 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:50.036932    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a783cfd9-8be0-4c36-9584-2fb86d0db268-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-6kbxk\" (UID: \"a783cfd9-8be0-4c36-9584-2fb86d0db268\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6kbxk"
Sep 18 14:03:50.036991 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:50.036952    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a783cfd9-8be0-4c36-9584-2fb86d0db268-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-6kbxk\" (UID: \"a783cfd9-8be0-4c36-9584-2fb86d0db268\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6kbxk"
Sep 18 14:03:50.036991 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:50.036973    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a783cfd9-8be0-4c36-9584-2fb86d0db268-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-6kbxk\" (UID: \"a783cfd9-8be0-4c36-9584-2fb86d0db268\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6kbxk"
Sep 18 14:03:50.049731 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:50.037017    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a783cfd9-8be0-4c36-9584-2fb86d0db268-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-6kbxk\" (UID: \"a783cfd9-8be0-4c36-9584-2fb86d0db268\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6kbxk"
Sep 18 14:03:50.049731 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:50.037044    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5p9p\" (UniqueName: \"kubernetes.io/projected/a783cfd9-8be0-4c36-9584-2fb86d0db268-kube-api-access-z5p9p\") pod \"istiod-openshift-gateway-7b7995d5cd-6kbxk\" (UID: \"a783cfd9-8be0-4c36-9584-2fb86d0db268\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6kbxk"
Sep 18 14:03:50.049731 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:50.038984    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a783cfd9-8be0-4c36-9584-2fb86d0db268-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-6kbxk\" (UID: \"a783cfd9-8be0-4c36-9584-2fb86d0db268\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6kbxk"
Sep 18 14:03:50.049731 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:50.039287    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 18 14:03:50.049731 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:50.039287    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 18 14:03:50.049731 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:50.039287    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 18 14:03:50.049731 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:50.039542    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 18 14:03:50.049731 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:50.045036    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a783cfd9-8be0-4c36-9584-2fb86d0db268-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-6kbxk\" (UID: \"a783cfd9-8be0-4c36-9584-2fb86d0db268\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6kbxk"
Sep 18 14:03:50.049731 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:50.045039    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5p9p\" (UniqueName: \"kubernetes.io/projected/a783cfd9-8be0-4c36-9584-2fb86d0db268-kube-api-access-z5p9p\") pod \"istiod-openshift-gateway-7b7995d5cd-6kbxk\" (UID: \"a783cfd9-8be0-4c36-9584-2fb86d0db268\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6kbxk"
Sep 18 14:03:50.049731 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:50.048279    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a783cfd9-8be0-4c36-9584-2fb86d0db268-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-6kbxk\" (UID: \"a783cfd9-8be0-4c36-9584-2fb86d0db268\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6kbxk"
Sep 18 14:03:50.050194 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:50.049743    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a783cfd9-8be0-4c36-9584-2fb86d0db268-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-6kbxk\" (UID: \"a783cfd9-8be0-4c36-9584-2fb86d0db268\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6kbxk"
Sep 18 14:03:50.050194 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:50.050020    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a783cfd9-8be0-4c36-9584-2fb86d0db268-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-6kbxk\" (UID: \"a783cfd9-8be0-4c36-9584-2fb86d0db268\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6kbxk"
Sep 18 14:03:50.050324 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:50.050305    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a783cfd9-8be0-4c36-9584-2fb86d0db268-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-6kbxk\" (UID: \"a783cfd9-8be0-4c36-9584-2fb86d0db268\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6kbxk"
Sep 18 14:03:50.166215 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:50.166140    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-86dc75d4b5-ls4xg" event={"ID":"6846cc5d-9e5f-4a5a-abbb-54c7163c1f96","Type":"ContainerStarted","Data":"26c16ff527941879c425438a588b5e0cebebe89a6d0d5dd8773fe0f73fc52831"}
Sep 18 14:03:50.191945 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:50.191923    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-hpqqd\""
Sep 18 14:03:50.200128 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:50.200095    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6kbxk"
Sep 18 14:03:50.520481 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:50.520367    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6kbxk"]
Sep 18 14:03:50.523782 ip-10-0-143-120 kubenswrapper[2579]: W0918 14:03:50.523747    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda783cfd9_8be0_4c36_9584_2fb86d0db268.slice/crio-e3963a64d1553680c7fad70c4a0659979acabf333097cae5a768ce78690035f7 WatchSource:0}: Error finding container e3963a64d1553680c7fad70c4a0659979acabf333097cae5a768ce78690035f7: Status 404 returned error can't find the container with id e3963a64d1553680c7fad70c4a0659979acabf333097cae5a768ce78690035f7
Sep 18 14:03:50.699413 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:50.699381    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:50.782967 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:50.782891    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:51.172091 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:51.172049    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6kbxk" event={"ID":"a783cfd9-8be0-4c36-9584-2fb86d0db268","Type":"ContainerStarted","Data":"e3963a64d1553680c7fad70c4a0659979acabf333097cae5a768ce78690035f7"}
Sep 18 14:03:51.273541 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:51.273511    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:53.522981 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:53.522957    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 18 14:03:53.566084 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:53.566050    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:03:53.566153 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:53.566126    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:03:54.187208 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:54.187154    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6kbxk" event={"ID":"a783cfd9-8be0-4c36-9584-2fb86d0db268","Type":"ContainerStarted","Data":"bd671b7f974641ae8e9ab3cd02596509570c7cddd913bfc9c6c74eb96590059a"}
Sep 18 14:03:54.187399 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:54.187234    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6kbxk"
Sep 18 14:03:54.188770 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:54.188740    2579 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-6kbxk container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 14:03:54.188887 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:54.188799    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6kbxk" podUID="a783cfd9-8be0-4c36-9584-2fb86d0db268" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 14:03:54.189060 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:54.189021    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-86dc75d4b5-ls4xg" event={"ID":"6846cc5d-9e5f-4a5a-abbb-54c7163c1f96","Type":"ContainerStarted","Data":"8c7e3a724bec7e24f95b201ae0326ab37433ca76e760f1b17fc30254de12c2c5"}
Sep 18 14:03:54.210922 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:54.210859    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6kbxk" podStartSLOduration=2.171439758 podStartE2EDuration="5.210846067s" totalImagesPullingTime="3.039406309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:03:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:03:54.209298496 +0000 UTC m=+306.098014427" watchObservedRunningTime="2026-09-18 14:03:54.210846067 +0000 UTC m=+306.099561999"
Sep 18 14:03:54.227046 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:54.227007    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-86dc75d4b5-ls4xg" podStartSLOduration=2.256383001 podStartE2EDuration="6.226992216s" totalImagesPullingTime="3.970609215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:03:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:03:54.22606776 +0000 UTC m=+306.114783692" watchObservedRunningTime="2026-09-18 14:03:54.226992216 +0000 UTC m=+306.115708148"
Sep 18 14:03:54.375632 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:54.375585    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:54.473941 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:54.473862    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:55.195554 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:55.195521    2579 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-6kbxk container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 14:03:55.195977 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:55.195579    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6kbxk" podUID="a783cfd9-8be0-4c36-9584-2fb86d0db268" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 14:03:55.555237 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:55.555206    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:55.708960 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:55.708934    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:55.865086 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:55.865023    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:58.196113 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:58.196078    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6kbxk"
Sep 18 14:03:59.146853 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:59.146814    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:59.223814 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:59.223763    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:59.410594 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:59.410520    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:03:59.712869 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:03:59.712794    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:04:00.250609 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:00.250578    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:04:05.297372 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:05.297330    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:04:06.400116 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:06.400083    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-vklds"]
Sep 18 14:04:06.403258 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:06.403243    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vklds"
Sep 18 14:04:06.435441 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:06.435422    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-vklds"]
Sep 18 14:04:06.453357 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:06.453330    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr2bd\" (UniqueName: \"kubernetes.io/projected/e4574081-5495-4cd9-8c39-0c7cded44228-kube-api-access-dr2bd\") pod \"istiod-openshift-gateway-5677ccb49b-vklds\" (UID: \"e4574081-5495-4cd9-8c39-0c7cded44228\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vklds"
Sep 18 14:04:06.453465 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:06.453362    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4574081-5495-4cd9-8c39-0c7cded44228-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-vklds\" (UID: \"e4574081-5495-4cd9-8c39-0c7cded44228\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vklds"
Sep 18 14:04:06.453465 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:06.453385    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e4574081-5495-4cd9-8c39-0c7cded44228-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-vklds\" (UID: \"e4574081-5495-4cd9-8c39-0c7cded44228\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vklds"
Sep 18 14:04:06.453465 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:06.453402    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e4574081-5495-4cd9-8c39-0c7cded44228-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-vklds\" (UID: \"e4574081-5495-4cd9-8c39-0c7cded44228\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vklds"
Sep 18 14:04:06.453595 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:06.453481    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4574081-5495-4cd9-8c39-0c7cded44228-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-vklds\" (UID: \"e4574081-5495-4cd9-8c39-0c7cded44228\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vklds"
Sep 18 14:04:06.453595 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:06.453498    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e4574081-5495-4cd9-8c39-0c7cded44228-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-vklds\" (UID: \"e4574081-5495-4cd9-8c39-0c7cded44228\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vklds"
Sep 18 14:04:06.453595 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:06.453520    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e4574081-5495-4cd9-8c39-0c7cded44228-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-vklds\" (UID: \"e4574081-5495-4cd9-8c39-0c7cded44228\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vklds"
Sep 18 14:04:06.553774 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:06.553749    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4574081-5495-4cd9-8c39-0c7cded44228-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-vklds\" (UID: \"e4574081-5495-4cd9-8c39-0c7cded44228\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vklds"
Sep 18 14:04:06.553885 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:06.553780    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e4574081-5495-4cd9-8c39-0c7cded44228-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-vklds\" (UID: \"e4574081-5495-4cd9-8c39-0c7cded44228\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vklds"
Sep 18 14:04:06.553885 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:06.553798    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e4574081-5495-4cd9-8c39-0c7cded44228-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-vklds\" (UID: \"e4574081-5495-4cd9-8c39-0c7cded44228\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vklds"
Sep 18 14:04:06.553885 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:06.553832    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4574081-5495-4cd9-8c39-0c7cded44228-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-vklds\" (UID: \"e4574081-5495-4cd9-8c39-0c7cded44228\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vklds"
Sep 18 14:04:06.554028 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:06.553970    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e4574081-5495-4cd9-8c39-0c7cded44228-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-vklds\" (UID: \"e4574081-5495-4cd9-8c39-0c7cded44228\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vklds"
Sep 18 14:04:06.554082 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:06.554033    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e4574081-5495-4cd9-8c39-0c7cded44228-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-vklds\" (UID: \"e4574081-5495-4cd9-8c39-0c7cded44228\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vklds"
Sep 18 14:04:06.554137 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:06.554082    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr2bd\" (UniqueName: \"kubernetes.io/projected/e4574081-5495-4cd9-8c39-0c7cded44228-kube-api-access-dr2bd\") pod \"istiod-openshift-gateway-5677ccb49b-vklds\" (UID: \"e4574081-5495-4cd9-8c39-0c7cded44228\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vklds"
Sep 18 14:04:06.554647 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:06.554624    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e4574081-5495-4cd9-8c39-0c7cded44228-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-vklds\" (UID: \"e4574081-5495-4cd9-8c39-0c7cded44228\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vklds"
Sep 18 14:04:06.556324 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:06.556296    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e4574081-5495-4cd9-8c39-0c7cded44228-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-vklds\" (UID: \"e4574081-5495-4cd9-8c39-0c7cded44228\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vklds"
Sep 18 14:04:06.556431 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:06.556412    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e4574081-5495-4cd9-8c39-0c7cded44228-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-vklds\" (UID: \"e4574081-5495-4cd9-8c39-0c7cded44228\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vklds"
Sep 18 14:04:06.556520 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:06.556503    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4574081-5495-4cd9-8c39-0c7cded44228-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-vklds\" (UID: \"e4574081-5495-4cd9-8c39-0c7cded44228\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vklds"
Sep 18 14:04:06.556577 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:06.556542    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e4574081-5495-4cd9-8c39-0c7cded44228-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-vklds\" (UID: \"e4574081-5495-4cd9-8c39-0c7cded44228\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vklds"
Sep 18 14:04:06.562265 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:06.562239    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4574081-5495-4cd9-8c39-0c7cded44228-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-vklds\" (UID: \"e4574081-5495-4cd9-8c39-0c7cded44228\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vklds"
Sep 18 14:04:06.562353 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:06.562267    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr2bd\" (UniqueName: \"kubernetes.io/projected/e4574081-5495-4cd9-8c39-0c7cded44228-kube-api-access-dr2bd\") pod \"istiod-openshift-gateway-5677ccb49b-vklds\" (UID: \"e4574081-5495-4cd9-8c39-0c7cded44228\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vklds"
Sep 18 14:04:06.712211 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:06.712151    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vklds"
Sep 18 14:04:07.030325 ip-10-0-143-120 kubenswrapper[2579]: W0918 14:04:07.030285    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4574081_5495_4cd9_8c39_0c7cded44228.slice/crio-8f753ef25347fc7bf497356973830479b91583c9a68ec15155cfef805a438862 WatchSource:0}: Error finding container 8f753ef25347fc7bf497356973830479b91583c9a68ec15155cfef805a438862: Status 404 returned error can't find the container with id 8f753ef25347fc7bf497356973830479b91583c9a68ec15155cfef805a438862
Sep 18 14:04:07.033457 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:07.033424    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:04:07.033562 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:07.033488    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:04:07.037240 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:07.037212    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-vklds"]
Sep 18 14:04:07.239095 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:07.239062    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vklds" event={"ID":"e4574081-5495-4cd9-8c39-0c7cded44228","Type":"ContainerStarted","Data":"a896ec5a8466bb161b4f5e4870a75a7d908474f5b30d825d55a31b380a629ef8"}
Sep 18 14:04:07.239255 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:07.239117    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vklds" event={"ID":"e4574081-5495-4cd9-8c39-0c7cded44228","Type":"ContainerStarted","Data":"8f753ef25347fc7bf497356973830479b91583c9a68ec15155cfef805a438862"}
Sep 18 14:04:07.239255 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:07.239218    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vklds"
Sep 18 14:04:07.277428 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:07.277389    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vklds" podStartSLOduration=1.277375766 podStartE2EDuration="1.277375766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:04:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:04:07.276266807 +0000 UTC m=+319.164982740" watchObservedRunningTime="2026-09-18 14:04:07.277375766 +0000 UTC m=+319.166091698"
Sep 18 14:04:08.244624 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:08.244598    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vklds"
Sep 18 14:04:08.327709 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:08.327674    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6kbxk"]
Sep 18 14:04:08.327887 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:08.327864    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6kbxk" podUID="a783cfd9-8be0-4c36-9584-2fb86d0db268" containerName="discovery" containerID="cri-o://bd671b7f974641ae8e9ab3cd02596509570c7cddd913bfc9c6c74eb96590059a" gracePeriod=30
Sep 18 14:04:08.556665 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:08.556639    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6kbxk"
Sep 18 14:04:08.671979 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:08.671957    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a783cfd9-8be0-4c36-9584-2fb86d0db268-istio-csr-dns-cert\") pod \"a783cfd9-8be0-4c36-9584-2fb86d0db268\" (UID: \"a783cfd9-8be0-4c36-9584-2fb86d0db268\") "
Sep 18 14:04:08.672081 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:08.671994    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a783cfd9-8be0-4c36-9584-2fb86d0db268-istio-token\") pod \"a783cfd9-8be0-4c36-9584-2fb86d0db268\" (UID: \"a783cfd9-8be0-4c36-9584-2fb86d0db268\") "
Sep 18 14:04:08.672498 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:08.672473    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a783cfd9-8be0-4c36-9584-2fb86d0db268-cacerts\") pod \"a783cfd9-8be0-4c36-9584-2fb86d0db268\" (UID: \"a783cfd9-8be0-4c36-9584-2fb86d0db268\") "
Sep 18 14:04:08.672717 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:08.672700    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a783cfd9-8be0-4c36-9584-2fb86d0db268-local-certs\") pod \"a783cfd9-8be0-4c36-9584-2fb86d0db268\" (UID: \"a783cfd9-8be0-4c36-9584-2fb86d0db268\") "
Sep 18 14:04:08.672844 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:08.672828    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a783cfd9-8be0-4c36-9584-2fb86d0db268-istio-kubeconfig\") pod \"a783cfd9-8be0-4c36-9584-2fb86d0db268\" (UID: \"a783cfd9-8be0-4c36-9584-2fb86d0db268\") "
Sep 18 14:04:08.672993 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:08.672978    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5p9p\" (UniqueName: \"kubernetes.io/projected/a783cfd9-8be0-4c36-9584-2fb86d0db268-kube-api-access-z5p9p\") pod \"a783cfd9-8be0-4c36-9584-2fb86d0db268\" (UID: \"a783cfd9-8be0-4c36-9584-2fb86d0db268\") "
Sep 18 14:04:08.673122 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:08.673106    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a783cfd9-8be0-4c36-9584-2fb86d0db268-istio-csr-ca-configmap\") pod \"a783cfd9-8be0-4c36-9584-2fb86d0db268\" (UID: \"a783cfd9-8be0-4c36-9584-2fb86d0db268\") "
Sep 18 14:04:08.673872 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:08.673837    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a783cfd9-8be0-4c36-9584-2fb86d0db268-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "a783cfd9-8be0-4c36-9584-2fb86d0db268" (UID: "a783cfd9-8be0-4c36-9584-2fb86d0db268"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:04:08.676731 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:08.676284    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a783cfd9-8be0-4c36-9584-2fb86d0db268-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "a783cfd9-8be0-4c36-9584-2fb86d0db268" (UID: "a783cfd9-8be0-4c36-9584-2fb86d0db268"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:04:08.676731 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:08.676341    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a783cfd9-8be0-4c36-9584-2fb86d0db268-cacerts" (OuterVolumeSpecName: "cacerts") pod "a783cfd9-8be0-4c36-9584-2fb86d0db268" (UID: "a783cfd9-8be0-4c36-9584-2fb86d0db268"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:04:08.678131 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:08.677379    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a783cfd9-8be0-4c36-9584-2fb86d0db268-local-certs" (OuterVolumeSpecName: "local-certs") pod "a783cfd9-8be0-4c36-9584-2fb86d0db268" (UID: "a783cfd9-8be0-4c36-9584-2fb86d0db268"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:04:08.678244 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:08.678207    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a783cfd9-8be0-4c36-9584-2fb86d0db268-istio-token" (OuterVolumeSpecName: "istio-token") pod "a783cfd9-8be0-4c36-9584-2fb86d0db268" (UID: "a783cfd9-8be0-4c36-9584-2fb86d0db268"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:04:08.679205 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:08.679022    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a783cfd9-8be0-4c36-9584-2fb86d0db268-kube-api-access-z5p9p" (OuterVolumeSpecName: "kube-api-access-z5p9p") pod "a783cfd9-8be0-4c36-9584-2fb86d0db268" (UID: "a783cfd9-8be0-4c36-9584-2fb86d0db268"). InnerVolumeSpecName "kube-api-access-z5p9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:04:08.679451 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:08.679421    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a783cfd9-8be0-4c36-9584-2fb86d0db268-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "a783cfd9-8be0-4c36-9584-2fb86d0db268" (UID: "a783cfd9-8be0-4c36-9584-2fb86d0db268"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:04:08.774081 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:08.774058    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5p9p\" (UniqueName: \"kubernetes.io/projected/a783cfd9-8be0-4c36-9584-2fb86d0db268-kube-api-access-z5p9p\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:04:08.774081 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:08.774078    2579 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a783cfd9-8be0-4c36-9584-2fb86d0db268-istio-csr-ca-configmap\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:04:08.774215 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:08.774088    2579 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a783cfd9-8be0-4c36-9584-2fb86d0db268-istio-csr-dns-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:04:08.774215 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:08.774097    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a783cfd9-8be0-4c36-9584-2fb86d0db268-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:04:08.774215 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:08.774105    2579 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a783cfd9-8be0-4c36-9584-2fb86d0db268-cacerts\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:04:08.774215 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:08.774112    2579 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a783cfd9-8be0-4c36-9584-2fb86d0db268-local-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:04:08.774215 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:08.774119    2579 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a783cfd9-8be0-4c36-9584-2fb86d0db268-istio-kubeconfig\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:04:09.248185 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:09.248146    2579 generic.go:356] "Generic (PLEG): container finished" podID="a783cfd9-8be0-4c36-9584-2fb86d0db268" containerID="bd671b7f974641ae8e9ab3cd02596509570c7cddd913bfc9c6c74eb96590059a" exitCode=0
Sep 18 14:04:09.248537 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:09.248241    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6kbxk"
Sep 18 14:04:09.248537 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:09.248237    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6kbxk" event={"ID":"a783cfd9-8be0-4c36-9584-2fb86d0db268","Type":"ContainerDied","Data":"bd671b7f974641ae8e9ab3cd02596509570c7cddd913bfc9c6c74eb96590059a"}
Sep 18 14:04:09.248537 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:09.248297    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6kbxk" event={"ID":"a783cfd9-8be0-4c36-9584-2fb86d0db268","Type":"ContainerDied","Data":"e3963a64d1553680c7fad70c4a0659979acabf333097cae5a768ce78690035f7"}
Sep 18 14:04:09.248537 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:09.248323    2579 scope.go:117] "RemoveContainer" containerID="bd671b7f974641ae8e9ab3cd02596509570c7cddd913bfc9c6c74eb96590059a"
Sep 18 14:04:09.257468 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:09.257447    2579 scope.go:117] "RemoveContainer" containerID="bd671b7f974641ae8e9ab3cd02596509570c7cddd913bfc9c6c74eb96590059a"
Sep 18 14:04:09.257809 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:04:09.257782    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd671b7f974641ae8e9ab3cd02596509570c7cddd913bfc9c6c74eb96590059a\": container with ID starting with bd671b7f974641ae8e9ab3cd02596509570c7cddd913bfc9c6c74eb96590059a not found: ID does not exist" containerID="bd671b7f974641ae8e9ab3cd02596509570c7cddd913bfc9c6c74eb96590059a"
Sep 18 14:04:09.257913 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:09.257814    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd671b7f974641ae8e9ab3cd02596509570c7cddd913bfc9c6c74eb96590059a"} err="failed to get container status \"bd671b7f974641ae8e9ab3cd02596509570c7cddd913bfc9c6c74eb96590059a\": rpc error: code = NotFound desc = could not find container \"bd671b7f974641ae8e9ab3cd02596509570c7cddd913bfc9c6c74eb96590059a\": container with ID starting with bd671b7f974641ae8e9ab3cd02596509570c7cddd913bfc9c6c74eb96590059a not found: ID does not exist"
Sep 18 14:04:09.284368 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:09.284347    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6kbxk"]
Sep 18 14:04:09.292972 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:09.292934    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6kbxk"]
Sep 18 14:04:09.395155 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:09.395128    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:04:09.468619 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:09.468592    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-l89sx"]
Sep 18 14:04:09.468829 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:09.468819    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a783cfd9-8be0-4c36-9584-2fb86d0db268" containerName="discovery"
Sep 18 14:04:09.468876 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:09.468832    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a783cfd9-8be0-4c36-9584-2fb86d0db268" containerName="discovery"
Sep 18 14:04:09.468910 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:09.468883    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a783cfd9-8be0-4c36-9584-2fb86d0db268" containerName="discovery"
Sep 18 14:04:09.472970 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:09.472955    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l89sx"
Sep 18 14:04:09.475829 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:09.475813    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-lsd4l\""
Sep 18 14:04:09.475894 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:09.475859    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 18 14:04:09.484518 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:09.484496    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-l89sx"]
Sep 18 14:04:09.490985 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:09.490961    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:04:09.578530 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:09.578509    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/413479c5-b502-4549-8b20-852bc9d3648f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-l89sx\" (UID: \"413479c5-b502-4549-8b20-852bc9d3648f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l89sx"
Sep 18 14:04:09.578624 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:09.578536    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjpnt\" (UniqueName: \"kubernetes.io/projected/413479c5-b502-4549-8b20-852bc9d3648f-kube-api-access-rjpnt\") pod \"llmisvc-controller-manager-777c6bd7d5-l89sx\" (UID: \"413479c5-b502-4549-8b20-852bc9d3648f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l89sx"
Sep 18 14:04:09.678819 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:09.678796    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/413479c5-b502-4549-8b20-852bc9d3648f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-l89sx\" (UID: \"413479c5-b502-4549-8b20-852bc9d3648f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l89sx"
Sep 18 14:04:09.678925 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:09.678831    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjpnt\" (UniqueName: \"kubernetes.io/projected/413479c5-b502-4549-8b20-852bc9d3648f-kube-api-access-rjpnt\") pod \"llmisvc-controller-manager-777c6bd7d5-l89sx\" (UID: \"413479c5-b502-4549-8b20-852bc9d3648f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l89sx"
Sep 18 14:04:09.679538 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:09.679508    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:04:09.681230 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:09.681212    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/413479c5-b502-4549-8b20-852bc9d3648f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-l89sx\" (UID: \"413479c5-b502-4549-8b20-852bc9d3648f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l89sx"
Sep 18 14:04:09.689668 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:09.689639    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjpnt\" (UniqueName: \"kubernetes.io/projected/413479c5-b502-4549-8b20-852bc9d3648f-kube-api-access-rjpnt\") pod \"llmisvc-controller-manager-777c6bd7d5-l89sx\" (UID: \"413479c5-b502-4549-8b20-852bc9d3648f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l89sx"
Sep 18 14:04:09.782474 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:09.782449    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l89sx"
Sep 18 14:04:09.830067 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:09.830031    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:04:10.013678 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:10.013647    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:04:10.034579 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:10.034556    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a783cfd9-8be0-4c36-9584-2fb86d0db268" path="/var/lib/kubelet/pods/a783cfd9-8be0-4c36-9584-2fb86d0db268/volumes"
Sep 18 14:04:10.095222 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:10.095160    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-l89sx"]
Sep 18 14:04:10.098255 ip-10-0-143-120 kubenswrapper[2579]: W0918 14:04:10.098233    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod413479c5_b502_4549_8b20_852bc9d3648f.slice/crio-0488d61021001c09eab9b04e488fa7c61c449d5c68c381cba6bd711c25ef2196 WatchSource:0}: Error finding container 0488d61021001c09eab9b04e488fa7c61c449d5c68c381cba6bd711c25ef2196: Status 404 returned error can't find the container with id 0488d61021001c09eab9b04e488fa7c61c449d5c68c381cba6bd711c25ef2196
Sep 18 14:04:10.099488 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:10.099474    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 14:04:10.252869 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:10.252841    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l89sx" event={"ID":"413479c5-b502-4549-8b20-852bc9d3648f","Type":"ContainerStarted","Data":"0488d61021001c09eab9b04e488fa7c61c449d5c68c381cba6bd711c25ef2196"}
Sep 18 14:04:13.267636 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:13.267607    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l89sx" event={"ID":"413479c5-b502-4549-8b20-852bc9d3648f","Type":"ContainerStarted","Data":"6a8812ea14a604f67ebd5c1a3f3eaad24e49ada160fdd8a16fc9310eee509df8"}
Sep 18 14:04:13.267981 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:13.267740    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l89sx"
Sep 18 14:04:13.288403 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:13.288363    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l89sx" podStartSLOduration=1.250414318 podStartE2EDuration="4.288349811s" totalImagesPullingTime="3.037935493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:04:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:04:13.287106012 +0000 UTC m=+325.175821957" watchObservedRunningTime="2026-09-18 14:04:13.288349811 +0000 UTC m=+325.177065778"
Sep 18 14:04:44.273593 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:44.273512    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l89sx"
Sep 18 14:04:46.078601 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:46.078566    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:04:59.936192 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:04:59.936123    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:00.031869 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:00.031827    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:17.909270 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:17.909234    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:18.999704 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:18.999659    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:21.181380 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:21.181347    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-c8cdh"]
Sep 18 14:05:21.184545 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:21.184523    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-c8cdh"
Sep 18 14:05:21.187499 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:21.187455    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 18 14:05:21.187681 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:21.187658    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-rt7j2\""
Sep 18 14:05:21.188311 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:21.188292    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 18 14:05:21.198003 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:21.197976    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-c8cdh"]
Sep 18 14:05:21.368189 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:21.368154    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:21.368364 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:21.368346    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdcll\" (UniqueName: \"kubernetes.io/projected/769de1a3-059d-4206-92e2-83379db6da05-kube-api-access-sdcll\") pod \"authorino-operator-76b478bd78-c8cdh\" (UID: \"769de1a3-059d-4206-92e2-83379db6da05\") " pod="kuadrant-system/authorino-operator-76b478bd78-c8cdh"
Sep 18 14:05:21.469260 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:21.469210    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdcll\" (UniqueName: \"kubernetes.io/projected/769de1a3-059d-4206-92e2-83379db6da05-kube-api-access-sdcll\") pod \"authorino-operator-76b478bd78-c8cdh\" (UID: \"769de1a3-059d-4206-92e2-83379db6da05\") " pod="kuadrant-system/authorino-operator-76b478bd78-c8cdh"
Sep 18 14:05:21.475809 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:21.475787    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:21.485875 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:21.485848    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdcll\" (UniqueName: \"kubernetes.io/projected/769de1a3-059d-4206-92e2-83379db6da05-kube-api-access-sdcll\") pod \"authorino-operator-76b478bd78-c8cdh\" (UID: \"769de1a3-059d-4206-92e2-83379db6da05\") " pod="kuadrant-system/authorino-operator-76b478bd78-c8cdh"
Sep 18 14:05:21.494676 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:21.494656    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-c8cdh"
Sep 18 14:05:21.782140 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:21.782118    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-c8cdh"]
Sep 18 14:05:21.783849 ip-10-0-143-120 kubenswrapper[2579]: W0918 14:05:21.783823    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod769de1a3_059d_4206_92e2_83379db6da05.slice/crio-2e001f9e389466868c74ad0cf342a7312aedbfbf3ccb45c9ef46f87b39ae5cae WatchSource:0}: Error finding container 2e001f9e389466868c74ad0cf342a7312aedbfbf3ccb45c9ef46f87b39ae5cae: Status 404 returned error can't find the container with id 2e001f9e389466868c74ad0cf342a7312aedbfbf3ccb45c9ef46f87b39ae5cae
Sep 18 14:05:22.126900 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:22.126873    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:22.524483 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:22.524389    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-c8cdh" event={"ID":"769de1a3-059d-4206-92e2-83379db6da05","Type":"ContainerStarted","Data":"2e001f9e389466868c74ad0cf342a7312aedbfbf3ccb45c9ef46f87b39ae5cae"}
Sep 18 14:05:23.577566 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:23.577508    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:24.117603 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:24.117574    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:24.415495 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:24.415473    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:24.520383 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:24.520350    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:24.534128 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:24.534105    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-c8cdh" event={"ID":"769de1a3-059d-4206-92e2-83379db6da05","Type":"ContainerStarted","Data":"e7d8dc6a5ca6828733eca3a3f99f87708a3b44bb7e481dc65c0a48a3cfaf3c3d"}
Sep 18 14:05:24.534242 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:24.534165    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-c8cdh"
Sep 18 14:05:24.554438 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:24.554397    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-c8cdh" podStartSLOduration=0.941199804 podStartE2EDuration="3.554383787s" totalImagesPullingTime="2.613183983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:05:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:05:24.553230215 +0000 UTC m=+396.441946146" watchObservedRunningTime="2026-09-18 14:05:24.554383787 +0000 UTC m=+396.443099718"
Sep 18 14:05:24.620250 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:24.620226    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:24.923479 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:24.923453    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:25.032346 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:25.032325    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:28.312927 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:28.312894    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:28.567082 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:28.567002    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:29.986075 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:29.986037    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-m7gl9"]
Sep 18 14:05:29.989153 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:29.989136    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-m7gl9"
Sep 18 14:05:29.993679 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:29.993656    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 18 14:05:29.993783 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:29.993667    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-qn5qr\""
Sep 18 14:05:30.007365 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:30.007343    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-m7gl9"]
Sep 18 14:05:30.030059 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:30.030031    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtx58\" (UniqueName: \"kubernetes.io/projected/9e657c79-304e-412c-88b5-9f694e8fae47-kube-api-access-wtx58\") pod \"dns-operator-controller-manager-7b84cd7c9c-m7gl9\" (UID: \"9e657c79-304e-412c-88b5-9f694e8fae47\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-m7gl9"
Sep 18 14:05:30.130699 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:30.130673    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtx58\" (UniqueName: \"kubernetes.io/projected/9e657c79-304e-412c-88b5-9f694e8fae47-kube-api-access-wtx58\") pod \"dns-operator-controller-manager-7b84cd7c9c-m7gl9\" (UID: \"9e657c79-304e-412c-88b5-9f694e8fae47\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-m7gl9"
Sep 18 14:05:30.140100 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:30.140074    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtx58\" (UniqueName: \"kubernetes.io/projected/9e657c79-304e-412c-88b5-9f694e8fae47-kube-api-access-wtx58\") pod \"dns-operator-controller-manager-7b84cd7c9c-m7gl9\" (UID: \"9e657c79-304e-412c-88b5-9f694e8fae47\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-m7gl9"
Sep 18 14:05:30.300012 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:30.299993    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-m7gl9"
Sep 18 14:05:30.587272 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:30.587247    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-m7gl9"]
Sep 18 14:05:30.589921 ip-10-0-143-120 kubenswrapper[2579]: W0918 14:05:30.589895    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e657c79_304e_412c_88b5_9f694e8fae47.slice/crio-f26f1b8ee68ed6150e97947f30503e35d7b920d7150cefa3cce347977125de61 WatchSource:0}: Error finding container f26f1b8ee68ed6150e97947f30503e35d7b920d7150cefa3cce347977125de61: Status 404 returned error can't find the container with id f26f1b8ee68ed6150e97947f30503e35d7b920d7150cefa3cce347977125de61
Sep 18 14:05:31.317454 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:31.317423    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:31.380101 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:31.380067    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:31.565496 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:31.565471    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-m7gl9" event={"ID":"9e657c79-304e-412c-88b5-9f694e8fae47","Type":"ContainerStarted","Data":"f26f1b8ee68ed6150e97947f30503e35d7b920d7150cefa3cce347977125de61"}
Sep 18 14:05:31.964808 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:31.964763    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:32.063072 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:32.063042    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:33.575144 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:33.575108    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-m7gl9" event={"ID":"9e657c79-304e-412c-88b5-9f694e8fae47","Type":"ContainerStarted","Data":"ee4ac0f0dadb8af285199e344aa85bcfa2d6c1993d9abfd1c69bb8eeaeb3a377"}
Sep 18 14:05:33.575528 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:33.575167    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-m7gl9"
Sep 18 14:05:33.592927 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:33.592871    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-m7gl9" podStartSLOduration=2.602638731 podStartE2EDuration="4.592856245s" totalImagesPullingTime="1.990217514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:05:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:05:33.591631258 +0000 UTC m=+405.480347192" watchObservedRunningTime="2026-09-18 14:05:33.592856245 +0000 UTC m=+405.481572228"
Sep 18 14:05:34.930709 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:34.930657    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:35.345602 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:35.345574    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:35.505305 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:35.505282    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:35.540918 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:35.540899    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-c8cdh"
Sep 18 14:05:35.609456 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:35.609396    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:39.948463 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:39.948427    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:40.044901 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:40.044870    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:40.145384 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:40.145357    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:43.840038 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:43.839998    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:43.943725 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:43.943694    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:44.581645 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:44.581607    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-m7gl9"
Sep 18 14:05:45.685183 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:45.685149    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xhkwr"]
Sep 18 14:05:45.688577 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:45.688558    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xhkwr"
Sep 18 14:05:45.693230 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:45.693214    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-q692n\""
Sep 18 14:05:45.713164 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:45.713135    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xhkwr"]
Sep 18 14:05:45.745266 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:45.745240    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9edcd61a-7653-4380-912f-cbd14a5bf79e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-xhkwr\" (UID: \"9edcd61a-7653-4380-912f-cbd14a5bf79e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xhkwr"
Sep 18 14:05:45.745363 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:45.745304    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6jwq\" (UniqueName: \"kubernetes.io/projected/9edcd61a-7653-4380-912f-cbd14a5bf79e-kube-api-access-f6jwq\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-xhkwr\" (UID: \"9edcd61a-7653-4380-912f-cbd14a5bf79e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xhkwr"
Sep 18 14:05:45.808351 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:45.808326    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xhkwr"]
Sep 18 14:05:45.808519 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:05:45.808501    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-f6jwq], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xhkwr" podUID="9edcd61a-7653-4380-912f-cbd14a5bf79e"
Sep 18 14:05:45.846206 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:45.846155    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6jwq\" (UniqueName: \"kubernetes.io/projected/9edcd61a-7653-4380-912f-cbd14a5bf79e-kube-api-access-f6jwq\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-xhkwr\" (UID: \"9edcd61a-7653-4380-912f-cbd14a5bf79e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xhkwr"
Sep 18 14:05:45.846308 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:45.846222    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9edcd61a-7653-4380-912f-cbd14a5bf79e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-xhkwr\" (UID: \"9edcd61a-7653-4380-912f-cbd14a5bf79e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xhkwr"
Sep 18 14:05:45.846631 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:45.846611    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9edcd61a-7653-4380-912f-cbd14a5bf79e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-xhkwr\" (UID: \"9edcd61a-7653-4380-912f-cbd14a5bf79e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xhkwr"
Sep 18 14:05:45.853379 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:45.853356    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7vlxm"]
Sep 18 14:05:45.855437 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:45.855424    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7vlxm"
Sep 18 14:05:45.860534 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:45.860515    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6jwq\" (UniqueName: \"kubernetes.io/projected/9edcd61a-7653-4380-912f-cbd14a5bf79e-kube-api-access-f6jwq\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-xhkwr\" (UID: \"9edcd61a-7653-4380-912f-cbd14a5bf79e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xhkwr"
Sep 18 14:05:45.873474 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:45.873453    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7vlxm"]
Sep 18 14:05:45.892001 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:45.891961    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7vlxm"]
Sep 18 14:05:45.892166 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:05:45.892146    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-kt6lk], unattached volumes=[], failed to process volumes=[extensions-socket-volume kube-api-access-kt6lk]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7vlxm" podUID="9a18a643-90c8-4f56-8b8c-269376a0e817"
Sep 18 14:05:45.903999 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:45.903976    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7vlxm"]
Sep 18 14:05:45.928778 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:45.928758    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xcjrd"]
Sep 18 14:05:45.930779 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:45.930764    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xcjrd"
Sep 18 14:05:45.951307 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:45.951284    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xcjrd"]
Sep 18 14:05:45.952466 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:45.952447    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xhkwr"]
Sep 18 14:05:45.961230 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:45.961164    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xhkwr"]
Sep 18 14:05:46.035622 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:46.035595    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a18a643-90c8-4f56-8b8c-269376a0e817" path="/var/lib/kubelet/pods/9a18a643-90c8-4f56-8b8c-269376a0e817/volumes"
Sep 18 14:05:46.047141 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:46.047120    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlszs\" (UniqueName: \"kubernetes.io/projected/c2973f4c-bada-49b8-a745-532a586bd54a-kube-api-access-qlszs\") pod \"kuadrant-operator-controller-manager-5769f99d9f-xcjrd\" (UID: \"c2973f4c-bada-49b8-a745-532a586bd54a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xcjrd"
Sep 18 14:05:46.047238 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:46.047163    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c2973f4c-bada-49b8-a745-532a586bd54a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-xcjrd\" (UID: \"c2973f4c-bada-49b8-a745-532a586bd54a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xcjrd"
Sep 18 14:05:46.148059 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:46.148032    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlszs\" (UniqueName: \"kubernetes.io/projected/c2973f4c-bada-49b8-a745-532a586bd54a-kube-api-access-qlszs\") pod \"kuadrant-operator-controller-manager-5769f99d9f-xcjrd\" (UID: \"c2973f4c-bada-49b8-a745-532a586bd54a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xcjrd"
Sep 18 14:05:46.148159 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:46.148079    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c2973f4c-bada-49b8-a745-532a586bd54a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-xcjrd\" (UID: \"c2973f4c-bada-49b8-a745-532a586bd54a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xcjrd"
Sep 18 14:05:46.148404 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:46.148389    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c2973f4c-bada-49b8-a745-532a586bd54a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-xcjrd\" (UID: \"c2973f4c-bada-49b8-a745-532a586bd54a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xcjrd"
Sep 18 14:05:46.163837 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:46.163815    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlszs\" (UniqueName: \"kubernetes.io/projected/c2973f4c-bada-49b8-a745-532a586bd54a-kube-api-access-qlszs\") pod \"kuadrant-operator-controller-manager-5769f99d9f-xcjrd\" (UID: \"c2973f4c-bada-49b8-a745-532a586bd54a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xcjrd"
Sep 18 14:05:46.239522 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:46.239460    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xcjrd"
Sep 18 14:05:46.479492 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:46.479440    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:46.541180 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:46.541148    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xcjrd"]
Sep 18 14:05:46.541308 ip-10-0-143-120 kubenswrapper[2579]: W0918 14:05:46.541255    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2973f4c_bada_49b8_a745_532a586bd54a.slice/crio-ac7136b2b794a37f21e11f750f8094175bb31836822dbf4203115fd77b190cc6 WatchSource:0}: Error finding container ac7136b2b794a37f21e11f750f8094175bb31836822dbf4203115fd77b190cc6: Status 404 returned error can't find the container with id ac7136b2b794a37f21e11f750f8094175bb31836822dbf4203115fd77b190cc6
Sep 18 14:05:46.631993 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:46.631972    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xhkwr"
Sep 18 14:05:46.632087 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:46.631991    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xcjrd" event={"ID":"c2973f4c-bada-49b8-a745-532a586bd54a","Type":"ContainerStarted","Data":"ac7136b2b794a37f21e11f750f8094175bb31836822dbf4203115fd77b190cc6"}
Sep 18 14:05:46.632129 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:46.632107    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7vlxm"
Sep 18 14:05:46.637770 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:46.637752    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xhkwr"
Sep 18 14:05:46.640231 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:46.640216    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7vlxm"
Sep 18 14:05:46.640363 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:05:46.640344    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-xhkwr\" is forbidden: User \"system:node:ip-10-0-143-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-120.ec2.internal' and this object" podUID="9edcd61a-7653-4380-912f-cbd14a5bf79e" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xhkwr"
Sep 18 14:05:46.643074 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:05:46.643056    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-xhkwr\" is forbidden: User \"system:node:ip-10-0-143-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-120.ec2.internal' and this object" podUID="9edcd61a-7653-4380-912f-cbd14a5bf79e" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xhkwr"
Sep 18 14:05:46.644921 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:05:46.644903    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5769f99d9f-7vlxm\" is forbidden: User \"system:node:ip-10-0-143-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-120.ec2.internal' and this object" podUID="9a18a643-90c8-4f56-8b8c-269376a0e817" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7vlxm"
Sep 18 14:05:46.675273 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:46.675251    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:46.751632 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:46.751610    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9edcd61a-7653-4380-912f-cbd14a5bf79e-extensions-socket-volume\") pod \"9edcd61a-7653-4380-912f-cbd14a5bf79e\" (UID: \"9edcd61a-7653-4380-912f-cbd14a5bf79e\") "
Sep 18 14:05:46.751942 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:46.751653    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f6jwq\" (UniqueName: \"kubernetes.io/projected/9edcd61a-7653-4380-912f-cbd14a5bf79e-kube-api-access-f6jwq\") pod \"9edcd61a-7653-4380-912f-cbd14a5bf79e\" (UID: \"9edcd61a-7653-4380-912f-cbd14a5bf79e\") "
Sep 18 14:05:46.751942 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:46.751850    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9edcd61a-7653-4380-912f-cbd14a5bf79e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9edcd61a-7653-4380-912f-cbd14a5bf79e" (UID: "9edcd61a-7653-4380-912f-cbd14a5bf79e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:05:46.753614 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:46.753596    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9edcd61a-7653-4380-912f-cbd14a5bf79e-kube-api-access-f6jwq" (OuterVolumeSpecName: "kube-api-access-f6jwq") pod "9edcd61a-7653-4380-912f-cbd14a5bf79e" (UID: "9edcd61a-7653-4380-912f-cbd14a5bf79e"). InnerVolumeSpecName "kube-api-access-f6jwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:05:46.852652 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:46.852623    2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9edcd61a-7653-4380-912f-cbd14a5bf79e-extensions-socket-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:05:46.852652 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:46.852653    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6jwq\" (UniqueName: \"kubernetes.io/projected/9edcd61a-7653-4380-912f-cbd14a5bf79e-kube-api-access-f6jwq\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:05:47.637521 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:47.637490    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7vlxm"
Sep 18 14:05:47.637700 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:47.637490    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xhkwr"
Sep 18 14:05:47.639947 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:05:47.639909    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-xhkwr\" is forbidden: User \"system:node:ip-10-0-143-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-120.ec2.internal' and this object" podUID="9edcd61a-7653-4380-912f-cbd14a5bf79e" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xhkwr"
Sep 18 14:05:47.641948 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:05:47.641910    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5769f99d9f-7vlxm\" is forbidden: User \"system:node:ip-10-0-143-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-120.ec2.internal' and this object" podUID="9a18a643-90c8-4f56-8b8c-269376a0e817" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7vlxm"
Sep 18 14:05:47.643614 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:05:47.643575    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-xhkwr\" is forbidden: User \"system:node:ip-10-0-143-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-120.ec2.internal' and this object" podUID="9edcd61a-7653-4380-912f-cbd14a5bf79e" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xhkwr"
Sep 18 14:05:47.645237 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:05:47.645212    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5769f99d9f-7vlxm\" is forbidden: User \"system:node:ip-10-0-143-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-120.ec2.internal' and this object" podUID="9a18a643-90c8-4f56-8b8c-269376a0e817" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7vlxm"
Sep 18 14:05:47.649230 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:05:47.649206    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-xhkwr\" is forbidden: User \"system:node:ip-10-0-143-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-120.ec2.internal' and this object" podUID="9edcd61a-7653-4380-912f-cbd14a5bf79e" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xhkwr"
Sep 18 14:05:47.650968 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:05:47.650943    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5769f99d9f-7vlxm\" is forbidden: User \"system:node:ip-10-0-143-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-120.ec2.internal' and this object" podUID="9a18a643-90c8-4f56-8b8c-269376a0e817" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7vlxm"
Sep 18 14:05:47.652530 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:05:47.652510    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-xhkwr\" is forbidden: User \"system:node:ip-10-0-143-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-120.ec2.internal' and this object" podUID="9edcd61a-7653-4380-912f-cbd14a5bf79e" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xhkwr"
Sep 18 14:05:47.654275 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:05:47.654248    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5769f99d9f-7vlxm\" is forbidden: User \"system:node:ip-10-0-143-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-120.ec2.internal' and this object" podUID="9a18a643-90c8-4f56-8b8c-269376a0e817" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7vlxm"
Sep 18 14:05:48.036792 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:48.036714    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9edcd61a-7653-4380-912f-cbd14a5bf79e" path="/var/lib/kubelet/pods/9edcd61a-7653-4380-912f-cbd14a5bf79e/volumes"
Sep 18 14:05:50.654022 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:50.653989    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xcjrd" event={"ID":"c2973f4c-bada-49b8-a745-532a586bd54a","Type":"ContainerStarted","Data":"693d26616f6d644ab0f256340ce43871a6139401c9909c2da6c8251252e58e93"}
Sep 18 14:05:50.654475 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:50.654119    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xcjrd"
Sep 18 14:05:50.677059 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:50.677001    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xcjrd" podStartSLOduration=2.14610481 podStartE2EDuration="5.67697876s" totalImagesPullingTime="3.53087395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:05:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:05:50.674685656 +0000 UTC m=+422.563401586" watchObservedRunningTime="2026-09-18 14:05:50.67697876 +0000 UTC m=+422.565694692"
Sep 18 14:05:51.958871 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:51.958820    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:52.016275 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:52.016244    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:52.556107 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:52.556064    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:52.656644 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:52.656610    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:55.965579 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:55.965541    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:05:56.066353 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:05:56.066306    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:06:00.159887 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:06:00.159856    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:06:00.263547 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:06:00.263519    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:06:00.366134 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:06:00.366106    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:06:00.424790 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:06:00.424721    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:06:00.570357 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:06:00.570332    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:06:00.671713 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:06:00.671690    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:06:01.661087 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:06:01.661059    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xcjrd"
Sep 18 14:06:02.657795 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:06:02.657719    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:06:02.756963 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:06:02.756931    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:06:06.559614 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:06:06.559580    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:06:06.660310 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:06:06.660279    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:06:18.105570 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:06:18.105537    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:06:18.317162 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:06:18.317130    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:06:20.038361 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:06:20.038320    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:06:20.142764 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:06:20.142729    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:06:20.241623 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:06:20.241593    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:06:20.877864 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:06:20.877829    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:06:20.974985 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:06:20.974956    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:06:20.979270 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:06:20.979241    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-9fws9"]
Sep 18 14:06:20.982380 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:06:20.982358    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-9fws9"
Sep 18 14:06:20.984765 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:06:20.984743    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-cbm25\""
Sep 18 14:06:20.991629 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:06:20.991610    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-9fws9"]
Sep 18 14:06:21.094513 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:06:21.094477    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl7bj\" (UniqueName: \"kubernetes.io/projected/9fa3ee8f-936a-4f70-8d0a-7fb59a2f81ea-kube-api-access-tl7bj\") pod \"authorino-59c974bd48-9fws9\" (UID: \"9fa3ee8f-936a-4f70-8d0a-7fb59a2f81ea\") " pod="kuadrant-system/authorino-59c974bd48-9fws9"
Sep 18 14:06:21.195549 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:06:21.195448    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl7bj\" (UniqueName: \"kubernetes.io/projected/9fa3ee8f-936a-4f70-8d0a-7fb59a2f81ea-kube-api-access-tl7bj\") pod \"authorino-59c974bd48-9fws9\" (UID: \"9fa3ee8f-936a-4f70-8d0a-7fb59a2f81ea\") " pod="kuadrant-system/authorino-59c974bd48-9fws9"
Sep 18 14:06:21.203895 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:06:21.203860    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl7bj\" (UniqueName: \"kubernetes.io/projected/9fa3ee8f-936a-4f70-8d0a-7fb59a2f81ea-kube-api-access-tl7bj\") pod \"authorino-59c974bd48-9fws9\" (UID: \"9fa3ee8f-936a-4f70-8d0a-7fb59a2f81ea\") " pod="kuadrant-system/authorino-59c974bd48-9fws9"
Sep 18 14:06:21.291934 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:06:21.291899    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-9fws9"
Sep 18 14:06:21.558358 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:06:21.558331    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-9fws9"]
Sep 18 14:06:21.561837 ip-10-0-143-120 kubenswrapper[2579]: W0918 14:06:21.561814    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9fa3ee8f_936a_4f70_8d0a_7fb59a2f81ea.slice/crio-07a3050be9bdbc2a589cff489b96ba8d00c06a1e811dfaeb3998f775a18d5990 WatchSource:0}: Error finding container 07a3050be9bdbc2a589cff489b96ba8d00c06a1e811dfaeb3998f775a18d5990: Status 404 returned error can't find the container with id 07a3050be9bdbc2a589cff489b96ba8d00c06a1e811dfaeb3998f775a18d5990
Sep 18 14:06:21.792734 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:06:21.792704    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-9fws9" event={"ID":"9fa3ee8f-936a-4f70-8d0a-7fb59a2f81ea","Type":"ContainerStarted","Data":"07a3050be9bdbc2a589cff489b96ba8d00c06a1e811dfaeb3998f775a18d5990"}
Sep 18 14:06:23.804151 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:06:23.804108    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-9fws9" event={"ID":"9fa3ee8f-936a-4f70-8d0a-7fb59a2f81ea","Type":"ContainerStarted","Data":"b1336c67d0c0fd2a59d3d048dc11e9930a454626068f454e197def5b2399f704"}
Sep 18 14:06:23.820577 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:06:23.820509    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-9fws9" podStartSLOduration=2.078818054 podStartE2EDuration="3.820485063s" totalImagesPullingTime="1.741667009s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:06:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:06:23.819639782 +0000 UTC m=+455.708355712" watchObservedRunningTime="2026-09-18 14:06:23.820485063 +0000 UTC m=+455.709200997"
Sep 18 14:06:41.385599 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:06:41.385562    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:06:41.609316 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:06:41.609284    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:06:41.805866 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:06:41.805835    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:06:54.572108 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:06:54.572075    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:06:55.143912 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:06:55.143885    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:06:56.893931 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:06:56.893896    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-9fws9"]
Sep 18 14:06:56.894316 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:06:56.894093    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-9fws9" podUID="9fa3ee8f-936a-4f70-8d0a-7fb59a2f81ea" containerName="authorino" containerID="cri-o://b1336c67d0c0fd2a59d3d048dc11e9930a454626068f454e197def5b2399f704" gracePeriod=30
Sep 18 14:06:57.135103 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:06:57.135081    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-9fws9"
Sep 18 14:06:57.146870 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:06:57.146827    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tl7bj\" (UniqueName: \"kubernetes.io/projected/9fa3ee8f-936a-4f70-8d0a-7fb59a2f81ea-kube-api-access-tl7bj\") pod \"9fa3ee8f-936a-4f70-8d0a-7fb59a2f81ea\" (UID: \"9fa3ee8f-936a-4f70-8d0a-7fb59a2f81ea\") "
Sep 18 14:06:57.148762 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:06:57.148741    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fa3ee8f-936a-4f70-8d0a-7fb59a2f81ea-kube-api-access-tl7bj" (OuterVolumeSpecName: "kube-api-access-tl7bj") pod "9fa3ee8f-936a-4f70-8d0a-7fb59a2f81ea" (UID: "9fa3ee8f-936a-4f70-8d0a-7fb59a2f81ea"). InnerVolumeSpecName "kube-api-access-tl7bj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:06:57.247780 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:06:57.247755    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl7bj\" (UniqueName: \"kubernetes.io/projected/9fa3ee8f-936a-4f70-8d0a-7fb59a2f81ea-kube-api-access-tl7bj\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:06:57.947956 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:06:57.947921    2579 generic.go:356] "Generic (PLEG): container finished" podID="9fa3ee8f-936a-4f70-8d0a-7fb59a2f81ea" containerID="b1336c67d0c0fd2a59d3d048dc11e9930a454626068f454e197def5b2399f704" exitCode=0
Sep 18 14:06:57.948366 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:06:57.947971    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-9fws9"
Sep 18 14:06:57.948366 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:06:57.948009    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-9fws9" event={"ID":"9fa3ee8f-936a-4f70-8d0a-7fb59a2f81ea","Type":"ContainerDied","Data":"b1336c67d0c0fd2a59d3d048dc11e9930a454626068f454e197def5b2399f704"}
Sep 18 14:06:57.948366 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:06:57.948059    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-9fws9" event={"ID":"9fa3ee8f-936a-4f70-8d0a-7fb59a2f81ea","Type":"ContainerDied","Data":"07a3050be9bdbc2a589cff489b96ba8d00c06a1e811dfaeb3998f775a18d5990"}
Sep 18 14:06:57.948366 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:06:57.948085    2579 scope.go:117] "RemoveContainer" containerID="b1336c67d0c0fd2a59d3d048dc11e9930a454626068f454e197def5b2399f704"
Sep 18 14:06:57.956940 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:06:57.956921    2579 scope.go:117] "RemoveContainer" containerID="b1336c67d0c0fd2a59d3d048dc11e9930a454626068f454e197def5b2399f704"
Sep 18 14:06:57.957272 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:06:57.957251    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1336c67d0c0fd2a59d3d048dc11e9930a454626068f454e197def5b2399f704\": container with ID starting with b1336c67d0c0fd2a59d3d048dc11e9930a454626068f454e197def5b2399f704 not found: ID does not exist" containerID="b1336c67d0c0fd2a59d3d048dc11e9930a454626068f454e197def5b2399f704"
Sep 18 14:06:57.957355 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:06:57.957280    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1336c67d0c0fd2a59d3d048dc11e9930a454626068f454e197def5b2399f704"} err="failed to get container status \"b1336c67d0c0fd2a59d3d048dc11e9930a454626068f454e197def5b2399f704\": rpc error: code = NotFound desc = could not find container \"b1336c67d0c0fd2a59d3d048dc11e9930a454626068f454e197def5b2399f704\": container with ID starting with b1336c67d0c0fd2a59d3d048dc11e9930a454626068f454e197def5b2399f704 not found: ID does not exist"
Sep 18 14:06:57.990737 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:06:57.990715    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-9fws9"]
Sep 18 14:06:58.002784 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:06:58.002760    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-9fws9"]
Sep 18 14:06:58.034999 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:06:58.034979    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fa3ee8f-936a-4f70-8d0a-7fb59a2f81ea" path="/var/lib/kubelet/pods/9fa3ee8f-936a-4f70-8d0a-7fb59a2f81ea/volumes"
Sep 18 14:07:01.608525 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:01.608496    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:07:01.640578 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:01.640556    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7568ffc469-l2vbl"]
Sep 18 14:07:01.640860 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:01.640845    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fa3ee8f-936a-4f70-8d0a-7fb59a2f81ea" containerName="authorino"
Sep 18 14:07:01.640860 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:01.640861    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fa3ee8f-936a-4f70-8d0a-7fb59a2f81ea" containerName="authorino"
Sep 18 14:07:01.640979 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:01.640942    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fa3ee8f-936a-4f70-8d0a-7fb59a2f81ea" containerName="authorino"
Sep 18 14:07:01.643527 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:01.643512    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7568ffc469-l2vbl"
Sep 18 14:07:01.650239 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:01.650222    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 18 14:07:01.650360 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:01.650303    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-svcgm\""
Sep 18 14:07:01.660287 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:01.660271    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 18 14:07:01.661451 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:01.661431    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 18 14:07:01.677697 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:01.677678    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtpp2\" (UniqueName: \"kubernetes.io/projected/bfbc3a65-2480-478f-b071-afa73012b0d9-kube-api-access-mtpp2\") pod \"maas-controller-7568ffc469-l2vbl\" (UID: \"bfbc3a65-2480-478f-b071-afa73012b0d9\") " pod="opendatahub/maas-controller-7568ffc469-l2vbl"
Sep 18 14:07:01.677782 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:01.677707    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfbc3a65-2480-478f-b071-afa73012b0d9-metrics-certs\") pod \"maas-controller-7568ffc469-l2vbl\" (UID: \"bfbc3a65-2480-478f-b071-afa73012b0d9\") " pod="opendatahub/maas-controller-7568ffc469-l2vbl"
Sep 18 14:07:01.677782 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:01.677731    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bfbc3a65-2480-478f-b071-afa73012b0d9-webhook-cert\") pod \"maas-controller-7568ffc469-l2vbl\" (UID: \"bfbc3a65-2480-478f-b071-afa73012b0d9\") " pod="opendatahub/maas-controller-7568ffc469-l2vbl"
Sep 18 14:07:01.708147 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:01.708126    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7568ffc469-l2vbl"]
Sep 18 14:07:01.719197 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:01.719153    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:07:01.778454 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:01.778433    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtpp2\" (UniqueName: \"kubernetes.io/projected/bfbc3a65-2480-478f-b071-afa73012b0d9-kube-api-access-mtpp2\") pod \"maas-controller-7568ffc469-l2vbl\" (UID: \"bfbc3a65-2480-478f-b071-afa73012b0d9\") " pod="opendatahub/maas-controller-7568ffc469-l2vbl"
Sep 18 14:07:01.778536 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:01.778460    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfbc3a65-2480-478f-b071-afa73012b0d9-metrics-certs\") pod \"maas-controller-7568ffc469-l2vbl\" (UID: \"bfbc3a65-2480-478f-b071-afa73012b0d9\") " pod="opendatahub/maas-controller-7568ffc469-l2vbl"
Sep 18 14:07:01.778536 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:01.778485    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bfbc3a65-2480-478f-b071-afa73012b0d9-webhook-cert\") pod \"maas-controller-7568ffc469-l2vbl\" (UID: \"bfbc3a65-2480-478f-b071-afa73012b0d9\") " pod="opendatahub/maas-controller-7568ffc469-l2vbl"
Sep 18 14:07:01.778604 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:07:01.778572    2579 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Sep 18 14:07:01.778677 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:07:01.778635    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfbc3a65-2480-478f-b071-afa73012b0d9-webhook-cert podName:bfbc3a65-2480-478f-b071-afa73012b0d9 nodeName:}" failed. No retries permitted until 2026-09-18 14:07:02.278611368 +0000 UTC m=+494.167327277 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/bfbc3a65-2480-478f-b071-afa73012b0d9-webhook-cert") pod "maas-controller-7568ffc469-l2vbl" (UID: "bfbc3a65-2480-478f-b071-afa73012b0d9") : secret "maas-controller-webhook-cert" not found
Sep 18 14:07:01.780976 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:01.780956    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfbc3a65-2480-478f-b071-afa73012b0d9-metrics-certs\") pod \"maas-controller-7568ffc469-l2vbl\" (UID: \"bfbc3a65-2480-478f-b071-afa73012b0d9\") " pod="opendatahub/maas-controller-7568ffc469-l2vbl"
Sep 18 14:07:01.799201 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:01.799161    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtpp2\" (UniqueName: \"kubernetes.io/projected/bfbc3a65-2480-478f-b071-afa73012b0d9-kube-api-access-mtpp2\") pod \"maas-controller-7568ffc469-l2vbl\" (UID: \"bfbc3a65-2480-478f-b071-afa73012b0d9\") " pod="opendatahub/maas-controller-7568ffc469-l2vbl"
Sep 18 14:07:01.850708 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:01.850673    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:07:01.924015 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:01.923963    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:07:02.282441 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:02.282385    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bfbc3a65-2480-478f-b071-afa73012b0d9-webhook-cert\") pod \"maas-controller-7568ffc469-l2vbl\" (UID: \"bfbc3a65-2480-478f-b071-afa73012b0d9\") " pod="opendatahub/maas-controller-7568ffc469-l2vbl"
Sep 18 14:07:02.284811 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:02.284787    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bfbc3a65-2480-478f-b071-afa73012b0d9-webhook-cert\") pod \"maas-controller-7568ffc469-l2vbl\" (UID: \"bfbc3a65-2480-478f-b071-afa73012b0d9\") " pod="opendatahub/maas-controller-7568ffc469-l2vbl"
Sep 18 14:07:02.553700 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:02.553670    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7568ffc469-l2vbl"
Sep 18 14:07:02.827875 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:02.827804    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7568ffc469-l2vbl"]
Sep 18 14:07:02.831783 ip-10-0-143-120 kubenswrapper[2579]: W0918 14:07:02.831758    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfbc3a65_2480_478f_b071_afa73012b0d9.slice/crio-5ab2becd958035c6f3d48209370d0e66413439ef95916a8615f3d3c6c29fb238 WatchSource:0}: Error finding container 5ab2becd958035c6f3d48209370d0e66413439ef95916a8615f3d3c6c29fb238: Status 404 returned error can't find the container with id 5ab2becd958035c6f3d48209370d0e66413439ef95916a8615f3d3c6c29fb238
Sep 18 14:07:02.975074 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:02.975046    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7568ffc469-l2vbl" event={"ID":"bfbc3a65-2480-478f-b071-afa73012b0d9","Type":"ContainerStarted","Data":"5ab2becd958035c6f3d48209370d0e66413439ef95916a8615f3d3c6c29fb238"}
Sep 18 14:07:05.042603 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:05.042567    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:07:05.145205 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:05.145157    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:07:05.989820 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:05.989785    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7568ffc469-l2vbl" event={"ID":"bfbc3a65-2480-478f-b071-afa73012b0d9","Type":"ContainerStarted","Data":"4101945a40991065bf57d4f7fa9dd43e7096ebbb4658a5da7769f353b1096a51"}
Sep 18 14:07:05.990087 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:05.989851    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7568ffc469-l2vbl"
Sep 18 14:07:06.009032 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:06.008988    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7568ffc469-l2vbl" podStartSLOduration=2.790110941 podStartE2EDuration="5.008969288s" totalImagesPullingTime="2.218858347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:07:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:07:06.007507468 +0000 UTC m=+497.896223424" watchObservedRunningTime="2026-09-18 14:07:06.008969288 +0000 UTC m=+497.897685219"
Sep 18 14:07:16.359994 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:16.359959    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7568ffc469-l2vbl"]
Sep 18 14:07:16.360439 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:16.360152    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7568ffc469-l2vbl" podUID="bfbc3a65-2480-478f-b071-afa73012b0d9" containerName="manager" containerID="cri-o://4101945a40991065bf57d4f7fa9dd43e7096ebbb4658a5da7769f353b1096a51" gracePeriod=30
Sep 18 14:07:16.361897 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:16.361868    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7568ffc469-l2vbl"
Sep 18 14:07:16.592731 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:16.592709    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7568ffc469-l2vbl"
Sep 18 14:07:16.682853 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:16.682788    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfbc3a65-2480-478f-b071-afa73012b0d9-metrics-certs\") pod \"bfbc3a65-2480-478f-b071-afa73012b0d9\" (UID: \"bfbc3a65-2480-478f-b071-afa73012b0d9\") "
Sep 18 14:07:16.682853 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:16.682835    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtpp2\" (UniqueName: \"kubernetes.io/projected/bfbc3a65-2480-478f-b071-afa73012b0d9-kube-api-access-mtpp2\") pod \"bfbc3a65-2480-478f-b071-afa73012b0d9\" (UID: \"bfbc3a65-2480-478f-b071-afa73012b0d9\") "
Sep 18 14:07:16.683005 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:16.682857    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bfbc3a65-2480-478f-b071-afa73012b0d9-webhook-cert\") pod \"bfbc3a65-2480-478f-b071-afa73012b0d9\" (UID: \"bfbc3a65-2480-478f-b071-afa73012b0d9\") "
Sep 18 14:07:16.684922 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:16.684887    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfbc3a65-2480-478f-b071-afa73012b0d9-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "bfbc3a65-2480-478f-b071-afa73012b0d9" (UID: "bfbc3a65-2480-478f-b071-afa73012b0d9"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:07:16.685022 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:16.684931    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfbc3a65-2480-478f-b071-afa73012b0d9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bfbc3a65-2480-478f-b071-afa73012b0d9" (UID: "bfbc3a65-2480-478f-b071-afa73012b0d9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:07:16.685022 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:16.684939    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfbc3a65-2480-478f-b071-afa73012b0d9-kube-api-access-mtpp2" (OuterVolumeSpecName: "kube-api-access-mtpp2") pod "bfbc3a65-2480-478f-b071-afa73012b0d9" (UID: "bfbc3a65-2480-478f-b071-afa73012b0d9"). InnerVolumeSpecName "kube-api-access-mtpp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:07:16.783585 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:16.783561    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtpp2\" (UniqueName: \"kubernetes.io/projected/bfbc3a65-2480-478f-b071-afa73012b0d9-kube-api-access-mtpp2\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:07:16.783585 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:16.783581    2579 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bfbc3a65-2480-478f-b071-afa73012b0d9-webhook-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:07:16.783709 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:16.783591    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfbc3a65-2480-478f-b071-afa73012b0d9-metrics-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:07:17.037574 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:17.037512    2579 generic.go:356] "Generic (PLEG): container finished" podID="bfbc3a65-2480-478f-b071-afa73012b0d9" containerID="4101945a40991065bf57d4f7fa9dd43e7096ebbb4658a5da7769f353b1096a51" exitCode=0
Sep 18 14:07:17.037574 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:17.037564    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7568ffc469-l2vbl"
Sep 18 14:07:17.037708 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:17.037601    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7568ffc469-l2vbl" event={"ID":"bfbc3a65-2480-478f-b071-afa73012b0d9","Type":"ContainerDied","Data":"4101945a40991065bf57d4f7fa9dd43e7096ebbb4658a5da7769f353b1096a51"}
Sep 18 14:07:17.037708 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:17.037655    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7568ffc469-l2vbl" event={"ID":"bfbc3a65-2480-478f-b071-afa73012b0d9","Type":"ContainerDied","Data":"5ab2becd958035c6f3d48209370d0e66413439ef95916a8615f3d3c6c29fb238"}
Sep 18 14:07:17.037708 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:17.037678    2579 scope.go:117] "RemoveContainer" containerID="4101945a40991065bf57d4f7fa9dd43e7096ebbb4658a5da7769f353b1096a51"
Sep 18 14:07:17.046846 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:17.046828    2579 scope.go:117] "RemoveContainer" containerID="4101945a40991065bf57d4f7fa9dd43e7096ebbb4658a5da7769f353b1096a51"
Sep 18 14:07:17.047142 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:07:17.047122    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4101945a40991065bf57d4f7fa9dd43e7096ebbb4658a5da7769f353b1096a51\": container with ID starting with 4101945a40991065bf57d4f7fa9dd43e7096ebbb4658a5da7769f353b1096a51 not found: ID does not exist" containerID="4101945a40991065bf57d4f7fa9dd43e7096ebbb4658a5da7769f353b1096a51"
Sep 18 14:07:17.047204 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:17.047152    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4101945a40991065bf57d4f7fa9dd43e7096ebbb4658a5da7769f353b1096a51"} err="failed to get container status \"4101945a40991065bf57d4f7fa9dd43e7096ebbb4658a5da7769f353b1096a51\": rpc error: code = NotFound desc = could not find container \"4101945a40991065bf57d4f7fa9dd43e7096ebbb4658a5da7769f353b1096a51\": container with ID starting with 4101945a40991065bf57d4f7fa9dd43e7096ebbb4658a5da7769f353b1096a51 not found: ID does not exist"
Sep 18 14:07:17.061045 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:17.061023    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7568ffc469-l2vbl"]
Sep 18 14:07:17.066498 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:17.066477    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7568ffc469-l2vbl"]
Sep 18 14:07:18.034925 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:18.034885    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfbc3a65-2480-478f-b071-afa73012b0d9" path="/var/lib/kubelet/pods/bfbc3a65-2480-478f-b071-afa73012b0d9/volumes"
Sep 18 14:07:37.250099 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:37.250023    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:07:37.347422 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:37.347398    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:07:38.700063 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:38.700031    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:07:38.800036 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:38.800010    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:07:38.953459 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:38.953391    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-55f6c99747-pv262"]
Sep 18 14:07:38.953707 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:38.953695    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfbc3a65-2480-478f-b071-afa73012b0d9" containerName="manager"
Sep 18 14:07:38.953749 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:38.953709    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfbc3a65-2480-478f-b071-afa73012b0d9" containerName="manager"
Sep 18 14:07:38.953783 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:38.953759    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfbc3a65-2480-478f-b071-afa73012b0d9" containerName="manager"
Sep 18 14:07:38.959180 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:38.959152    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55f6c99747-pv262"
Sep 18 14:07:38.961723 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:38.961695    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 18 14:07:38.961723 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:38.961708    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 18 14:07:38.961893 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:38.961702    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 14:07:38.961893 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:38.961805    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 18 14:07:38.962446 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:38.962430    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5nh5f\""
Sep 18 14:07:38.968827 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:38.968801    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-55f6c99747-pv262"]
Sep 18 14:07:39.042881 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:39.042856    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2c22a5f-b11c-416b-936c-ea0fd28c4614-metrics-certs\") pod \"maas-api-55f6c99747-pv262\" (UID: \"d2c22a5f-b11c-416b-936c-ea0fd28c4614\") " pod="odh-ai-gateway-infra/maas-api-55f6c99747-pv262"
Sep 18 14:07:39.042978 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:39.042884    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2c22a5f-b11c-416b-936c-ea0fd28c4614-openshift-service-ca\") pod \"maas-api-55f6c99747-pv262\" (UID: \"d2c22a5f-b11c-416b-936c-ea0fd28c4614\") " pod="odh-ai-gateway-infra/maas-api-55f6c99747-pv262"
Sep 18 14:07:39.042978 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:39.042904    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2c22a5f-b11c-416b-936c-ea0fd28c4614-maas-api-tls\") pod \"maas-api-55f6c99747-pv262\" (UID: \"d2c22a5f-b11c-416b-936c-ea0fd28c4614\") " pod="odh-ai-gateway-infra/maas-api-55f6c99747-pv262"
Sep 18 14:07:39.043050 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:39.042996    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9cmv\" (UniqueName: \"kubernetes.io/projected/d2c22a5f-b11c-416b-936c-ea0fd28c4614-kube-api-access-t9cmv\") pod \"maas-api-55f6c99747-pv262\" (UID: \"d2c22a5f-b11c-416b-936c-ea0fd28c4614\") " pod="odh-ai-gateway-infra/maas-api-55f6c99747-pv262"
Sep 18 14:07:39.129740 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:39.129709    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:07:39.143266 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:39.143244    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9cmv\" (UniqueName: \"kubernetes.io/projected/d2c22a5f-b11c-416b-936c-ea0fd28c4614-kube-api-access-t9cmv\") pod \"maas-api-55f6c99747-pv262\" (UID: \"d2c22a5f-b11c-416b-936c-ea0fd28c4614\") " pod="odh-ai-gateway-infra/maas-api-55f6c99747-pv262"
Sep 18 14:07:39.143362 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:39.143310    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2c22a5f-b11c-416b-936c-ea0fd28c4614-metrics-certs\") pod \"maas-api-55f6c99747-pv262\" (UID: \"d2c22a5f-b11c-416b-936c-ea0fd28c4614\") " pod="odh-ai-gateway-infra/maas-api-55f6c99747-pv262"
Sep 18 14:07:39.143362 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:39.143340    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2c22a5f-b11c-416b-936c-ea0fd28c4614-openshift-service-ca\") pod \"maas-api-55f6c99747-pv262\" (UID: \"d2c22a5f-b11c-416b-936c-ea0fd28c4614\") " pod="odh-ai-gateway-infra/maas-api-55f6c99747-pv262"
Sep 18 14:07:39.143477 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:39.143366    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2c22a5f-b11c-416b-936c-ea0fd28c4614-maas-api-tls\") pod \"maas-api-55f6c99747-pv262\" (UID: \"d2c22a5f-b11c-416b-936c-ea0fd28c4614\") " pod="odh-ai-gateway-infra/maas-api-55f6c99747-pv262"
Sep 18 14:07:39.143477 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:07:39.143458    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-metrics-tls: secret "maas-api-metrics-tls" not found
Sep 18 14:07:39.143570 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:07:39.143533    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2c22a5f-b11c-416b-936c-ea0fd28c4614-metrics-certs podName:d2c22a5f-b11c-416b-936c-ea0fd28c4614 nodeName:}" failed. No retries permitted until 2026-09-18 14:07:39.643513539 +0000 UTC m=+531.532229451 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2c22a5f-b11c-416b-936c-ea0fd28c4614-metrics-certs") pod "maas-api-55f6c99747-pv262" (UID: "d2c22a5f-b11c-416b-936c-ea0fd28c4614") : secret "maas-api-metrics-tls" not found
Sep 18 14:07:39.144049 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:39.144032    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2c22a5f-b11c-416b-936c-ea0fd28c4614-openshift-service-ca\") pod \"maas-api-55f6c99747-pv262\" (UID: \"d2c22a5f-b11c-416b-936c-ea0fd28c4614\") " pod="odh-ai-gateway-infra/maas-api-55f6c99747-pv262"
Sep 18 14:07:39.145710 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:39.145689    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2c22a5f-b11c-416b-936c-ea0fd28c4614-maas-api-tls\") pod \"maas-api-55f6c99747-pv262\" (UID: \"d2c22a5f-b11c-416b-936c-ea0fd28c4614\") " pod="odh-ai-gateway-infra/maas-api-55f6c99747-pv262"
Sep 18 14:07:39.151733 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:39.151715    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9cmv\" (UniqueName: \"kubernetes.io/projected/d2c22a5f-b11c-416b-936c-ea0fd28c4614-kube-api-access-t9cmv\") pod \"maas-api-55f6c99747-pv262\" (UID: \"d2c22a5f-b11c-416b-936c-ea0fd28c4614\") " pod="odh-ai-gateway-infra/maas-api-55f6c99747-pv262"
Sep 18 14:07:39.230600 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:39.230540    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:07:39.461909 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:39.461877    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:07:39.576730 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:39.576696    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:07:39.646943 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:39.646919    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2c22a5f-b11c-416b-936c-ea0fd28c4614-metrics-certs\") pod \"maas-api-55f6c99747-pv262\" (UID: \"d2c22a5f-b11c-416b-936c-ea0fd28c4614\") " pod="odh-ai-gateway-infra/maas-api-55f6c99747-pv262"
Sep 18 14:07:39.649123 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:39.649103    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2c22a5f-b11c-416b-936c-ea0fd28c4614-metrics-certs\") pod \"maas-api-55f6c99747-pv262\" (UID: \"d2c22a5f-b11c-416b-936c-ea0fd28c4614\") " pod="odh-ai-gateway-infra/maas-api-55f6c99747-pv262"
Sep 18 14:07:39.870742 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:39.870679    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55f6c99747-pv262"
Sep 18 14:07:40.142550 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:40.142398    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-55f6c99747-pv262"]
Sep 18 14:07:40.144850 ip-10-0-143-120 kubenswrapper[2579]: W0918 14:07:40.144828    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2c22a5f_b11c_416b_936c_ea0fd28c4614.slice/crio-e52a4314f6522a424b7b40468c28b55084f3f4139c6e84b6cbcf6d2b98a0b9dd WatchSource:0}: Error finding container e52a4314f6522a424b7b40468c28b55084f3f4139c6e84b6cbcf6d2b98a0b9dd: Status 404 returned error can't find the container with id e52a4314f6522a424b7b40468c28b55084f3f4139c6e84b6cbcf6d2b98a0b9dd
Sep 18 14:07:41.135193 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:41.135144    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55f6c99747-pv262" event={"ID":"d2c22a5f-b11c-416b-936c-ea0fd28c4614","Type":"ContainerStarted","Data":"e52a4314f6522a424b7b40468c28b55084f3f4139c6e84b6cbcf6d2b98a0b9dd"}
Sep 18 14:07:42.140627 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:42.140595    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55f6c99747-pv262" event={"ID":"d2c22a5f-b11c-416b-936c-ea0fd28c4614","Type":"ContainerStarted","Data":"7b0298d1aab5148f9001ec4e463a9d565222941605826313496fb0c4102f1754"}
Sep 18 14:07:42.140952 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:42.140707    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-55f6c99747-pv262"
Sep 18 14:07:42.157837 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:42.157798    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-55f6c99747-pv262" podStartSLOduration=2.318518554 podStartE2EDuration="4.15778454s" totalImagesPullingTime="1.839265986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:07:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:07:42.155991204 +0000 UTC m=+534.044707135" watchObservedRunningTime="2026-09-18 14:07:42.15778454 +0000 UTC m=+534.046500490"
Sep 18 14:07:48.155567 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:48.155529    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-55f6c99747-pv262"
Sep 18 14:07:49.159913 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:49.159884    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:07:49.256990 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:49.256961    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:07:57.251066 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:57.251030    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:07:57.352204 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:57.352155    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:07:58.347022 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:07:58.346983    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:18.121840 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:08:18.121809    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6gb2n"]
Sep 18 14:08:18.126663 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:08:18.126646    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6gb2n"
Sep 18 14:08:18.129056 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:08:18.129033    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 18 14:08:18.129165 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:08:18.129033    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-5zz6x\""
Sep 18 14:08:18.129786 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:08:18.129769    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 18 14:08:18.129875 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:08:18.129849    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 18 14:08:18.134023 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:08:18.134000    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6gb2n"]
Sep 18 14:08:18.158129 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:08:18.158108    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:18.206077 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:08:18.206055    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05d9cb17-23c2-41d1-aa47-3d7b7588a3bd-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6gb2n\" (UID: \"05d9cb17-23c2-41d1-aa47-3d7b7588a3bd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6gb2n"
Sep 18 14:08:18.206200 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:08:18.206091    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mffqx\" (UniqueName: \"kubernetes.io/projected/05d9cb17-23c2-41d1-aa47-3d7b7588a3bd-kube-api-access-mffqx\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6gb2n\" (UID: \"05d9cb17-23c2-41d1-aa47-3d7b7588a3bd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6gb2n"
Sep 18 14:08:18.206200 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:08:18.206108    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05d9cb17-23c2-41d1-aa47-3d7b7588a3bd-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6gb2n\" (UID: \"05d9cb17-23c2-41d1-aa47-3d7b7588a3bd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6gb2n"
Sep 18 14:08:18.206324 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:08:18.206221    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05d9cb17-23c2-41d1-aa47-3d7b7588a3bd-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6gb2n\" (UID: \"05d9cb17-23c2-41d1-aa47-3d7b7588a3bd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6gb2n"
Sep 18 14:08:18.206324 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:08:18.206273    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05d9cb17-23c2-41d1-aa47-3d7b7588a3bd-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6gb2n\" (UID: \"05d9cb17-23c2-41d1-aa47-3d7b7588a3bd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6gb2n"
Sep 18 14:08:18.206324 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:08:18.206301    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05d9cb17-23c2-41d1-aa47-3d7b7588a3bd-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6gb2n\" (UID: \"05d9cb17-23c2-41d1-aa47-3d7b7588a3bd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6gb2n"
Sep 18 14:08:18.306584 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:08:18.306556    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mffqx\" (UniqueName: \"kubernetes.io/projected/05d9cb17-23c2-41d1-aa47-3d7b7588a3bd-kube-api-access-mffqx\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6gb2n\" (UID: \"05d9cb17-23c2-41d1-aa47-3d7b7588a3bd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6gb2n"
Sep 18 14:08:18.306697 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:08:18.306585    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05d9cb17-23c2-41d1-aa47-3d7b7588a3bd-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6gb2n\" (UID: \"05d9cb17-23c2-41d1-aa47-3d7b7588a3bd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6gb2n"
Sep 18 14:08:18.306697 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:08:18.306616    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05d9cb17-23c2-41d1-aa47-3d7b7588a3bd-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6gb2n\" (UID: \"05d9cb17-23c2-41d1-aa47-3d7b7588a3bd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6gb2n"
Sep 18 14:08:18.306815 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:08:18.306743    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05d9cb17-23c2-41d1-aa47-3d7b7588a3bd-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6gb2n\" (UID: \"05d9cb17-23c2-41d1-aa47-3d7b7588a3bd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6gb2n"
Sep 18 14:08:18.306815 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:08:18.306774    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05d9cb17-23c2-41d1-aa47-3d7b7588a3bd-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6gb2n\" (UID: \"05d9cb17-23c2-41d1-aa47-3d7b7588a3bd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6gb2n"
Sep 18 14:08:18.306922 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:08:18.306815    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05d9cb17-23c2-41d1-aa47-3d7b7588a3bd-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6gb2n\" (UID: \"05d9cb17-23c2-41d1-aa47-3d7b7588a3bd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6gb2n"
Sep 18 14:08:18.307161 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:08:18.307141    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05d9cb17-23c2-41d1-aa47-3d7b7588a3bd-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6gb2n\" (UID: \"05d9cb17-23c2-41d1-aa47-3d7b7588a3bd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6gb2n"
Sep 18 14:08:18.307161 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:08:18.307155    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05d9cb17-23c2-41d1-aa47-3d7b7588a3bd-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6gb2n\" (UID: \"05d9cb17-23c2-41d1-aa47-3d7b7588a3bd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6gb2n"
Sep 18 14:08:18.307376 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:08:18.307237    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05d9cb17-23c2-41d1-aa47-3d7b7588a3bd-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6gb2n\" (UID: \"05d9cb17-23c2-41d1-aa47-3d7b7588a3bd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6gb2n"
Sep 18 14:08:18.308798 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:08:18.308780    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05d9cb17-23c2-41d1-aa47-3d7b7588a3bd-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6gb2n\" (UID: \"05d9cb17-23c2-41d1-aa47-3d7b7588a3bd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6gb2n"
Sep 18 14:08:18.309078 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:08:18.309061    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05d9cb17-23c2-41d1-aa47-3d7b7588a3bd-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6gb2n\" (UID: \"05d9cb17-23c2-41d1-aa47-3d7b7588a3bd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6gb2n"
Sep 18 14:08:18.314587 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:08:18.314564    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mffqx\" (UniqueName: \"kubernetes.io/projected/05d9cb17-23c2-41d1-aa47-3d7b7588a3bd-kube-api-access-mffqx\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6gb2n\" (UID: \"05d9cb17-23c2-41d1-aa47-3d7b7588a3bd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6gb2n"
Sep 18 14:08:18.438668 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:08:18.438614    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6gb2n"
Sep 18 14:08:18.718124 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:08:18.718098    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6gb2n"]
Sep 18 14:08:18.720634 ip-10-0-143-120 kubenswrapper[2579]: W0918 14:08:18.720609    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05d9cb17_23c2_41d1_aa47_3d7b7588a3bd.slice/crio-48a55043b25d60cc08d24f61e415ef969c25a7ee02c6c4d51524a71bea088e35 WatchSource:0}: Error finding container 48a55043b25d60cc08d24f61e415ef969c25a7ee02c6c4d51524a71bea088e35: Status 404 returned error can't find the container with id 48a55043b25d60cc08d24f61e415ef969c25a7ee02c6c4d51524a71bea088e35
Sep 18 14:08:19.299190 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:08:19.299136    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6gb2n" event={"ID":"05d9cb17-23c2-41d1-aa47-3d7b7588a3bd","Type":"ContainerStarted","Data":"48a55043b25d60cc08d24f61e415ef969c25a7ee02c6c4d51524a71bea088e35"}
Sep 18 14:08:20.304418 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:08:20.304392    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6gb2n" event={"ID":"05d9cb17-23c2-41d1-aa47-3d7b7588a3bd","Type":"ContainerStarted","Data":"37b0841e3ee62eb91b8583a746f1a0c57e1a1e6b6b8c2bd14f20c2a8bc807acd"}
Sep 18 14:08:20.324016 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:08:20.323966    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6gb2n" podStartSLOduration=1.244340379 podStartE2EDuration="2.323949755s" totalImagesPullingTime="1.079609376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:08:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:08:20.321924983 +0000 UTC m=+572.210640913" watchObservedRunningTime="2026-09-18 14:08:20.323949755 +0000 UTC m=+572.212665685"
Sep 18 14:08:28.438955 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:08:28.438925    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6gb2n"
Sep 18 14:08:28.439501 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:08:28.438978    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6gb2n"
Sep 18 14:08:28.451154 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:08:28.451126    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6gb2n"
Sep 18 14:08:29.349756 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:08:29.349730    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6gb2n"
Sep 18 14:08:39.812285 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:08:39.812254    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:08:49.956055 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:08:49.956025    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sg4tv_4626a667-4cd6-468c-8ef3-6e3a925c2608/ovn-acl-logging/0.log"
Sep 18 14:08:49.957018 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:08:49.956985    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sg4tv_4626a667-4cd6-468c-8ef3-6e3a925c2608/ovn-acl-logging/0.log"
Sep 18 14:09:16.020949 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:09:16.020918    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bj7wf"]
Sep 18 14:09:16.024085 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:09:16.024069    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bj7wf"
Sep 18 14:09:16.026831 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:09:16.026811    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 18 14:09:16.037231 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:09:16.037213    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bj7wf"]
Sep 18 14:09:16.056029 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:09:16.056000    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:16.124874 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:09:16.124852    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/901a17ec-cc37-47c1-bb69-29c51f62e87e-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-bj7wf\" (UID: \"901a17ec-cc37-47c1-bb69-29c51f62e87e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bj7wf"
Sep 18 14:09:16.124976 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:09:16.124878    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/901a17ec-cc37-47c1-bb69-29c51f62e87e-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-bj7wf\" (UID: \"901a17ec-cc37-47c1-bb69-29c51f62e87e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bj7wf"
Sep 18 14:09:16.124976 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:09:16.124897    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwt4j\" (UniqueName: \"kubernetes.io/projected/901a17ec-cc37-47c1-bb69-29c51f62e87e-kube-api-access-bwt4j\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-bj7wf\" (UID: \"901a17ec-cc37-47c1-bb69-29c51f62e87e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bj7wf"
Sep 18 14:09:16.124976 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:09:16.124920    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/901a17ec-cc37-47c1-bb69-29c51f62e87e-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-bj7wf\" (UID: \"901a17ec-cc37-47c1-bb69-29c51f62e87e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bj7wf"
Sep 18 14:09:16.125133 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:09:16.125014    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/901a17ec-cc37-47c1-bb69-29c51f62e87e-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-bj7wf\" (UID: \"901a17ec-cc37-47c1-bb69-29c51f62e87e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bj7wf"
Sep 18 14:09:16.125133 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:09:16.125065    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/901a17ec-cc37-47c1-bb69-29c51f62e87e-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-bj7wf\" (UID: \"901a17ec-cc37-47c1-bb69-29c51f62e87e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bj7wf"
Sep 18 14:09:16.225898 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:09:16.225869    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/901a17ec-cc37-47c1-bb69-29c51f62e87e-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-bj7wf\" (UID: \"901a17ec-cc37-47c1-bb69-29c51f62e87e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bj7wf"
Sep 18 14:09:16.226002 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:09:16.225900    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/901a17ec-cc37-47c1-bb69-29c51f62e87e-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-bj7wf\" (UID: \"901a17ec-cc37-47c1-bb69-29c51f62e87e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bj7wf"
Sep 18 14:09:16.226002 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:09:16.225916    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwt4j\" (UniqueName: \"kubernetes.io/projected/901a17ec-cc37-47c1-bb69-29c51f62e87e-kube-api-access-bwt4j\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-bj7wf\" (UID: \"901a17ec-cc37-47c1-bb69-29c51f62e87e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bj7wf"
Sep 18 14:09:16.226002 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:09:16.225938    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/901a17ec-cc37-47c1-bb69-29c51f62e87e-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-bj7wf\" (UID: \"901a17ec-cc37-47c1-bb69-29c51f62e87e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bj7wf"
Sep 18 14:09:16.226002 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:09:16.225960    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/901a17ec-cc37-47c1-bb69-29c51f62e87e-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-bj7wf\" (UID: \"901a17ec-cc37-47c1-bb69-29c51f62e87e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bj7wf"
Sep 18 14:09:16.226002 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:09:16.225982    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/901a17ec-cc37-47c1-bb69-29c51f62e87e-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-bj7wf\" (UID: \"901a17ec-cc37-47c1-bb69-29c51f62e87e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bj7wf"
Sep 18 14:09:16.226336 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:09:16.226314    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/901a17ec-cc37-47c1-bb69-29c51f62e87e-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-bj7wf\" (UID: \"901a17ec-cc37-47c1-bb69-29c51f62e87e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bj7wf"
Sep 18 14:09:16.226390 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:09:16.226362    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/901a17ec-cc37-47c1-bb69-29c51f62e87e-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-bj7wf\" (UID: \"901a17ec-cc37-47c1-bb69-29c51f62e87e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bj7wf"
Sep 18 14:09:16.226455 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:09:16.226438    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/901a17ec-cc37-47c1-bb69-29c51f62e87e-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-bj7wf\" (UID: \"901a17ec-cc37-47c1-bb69-29c51f62e87e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bj7wf"
Sep 18 14:09:16.228029 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:09:16.228008    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/901a17ec-cc37-47c1-bb69-29c51f62e87e-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-bj7wf\" (UID: \"901a17ec-cc37-47c1-bb69-29c51f62e87e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bj7wf"
Sep 18 14:09:16.228307 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:09:16.228292    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/901a17ec-cc37-47c1-bb69-29c51f62e87e-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-bj7wf\" (UID: \"901a17ec-cc37-47c1-bb69-29c51f62e87e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bj7wf"
Sep 18 14:09:16.250183 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:09:16.250149    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwt4j\" (UniqueName: \"kubernetes.io/projected/901a17ec-cc37-47c1-bb69-29c51f62e87e-kube-api-access-bwt4j\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-bj7wf\" (UID: \"901a17ec-cc37-47c1-bb69-29c51f62e87e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bj7wf"
Sep 18 14:09:16.335334 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:09:16.335303    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bj7wf"
Sep 18 14:09:16.606165 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:09:16.606143    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bj7wf"]
Sep 18 14:09:16.608366 ip-10-0-143-120 kubenswrapper[2579]: W0918 14:09:16.608343    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod901a17ec_cc37_47c1_bb69_29c51f62e87e.slice/crio-256153f3cb10d78d44e84f19d76efba928f65f326093b8e80290e5a794f9c942 WatchSource:0}: Error finding container 256153f3cb10d78d44e84f19d76efba928f65f326093b8e80290e5a794f9c942: Status 404 returned error can't find the container with id 256153f3cb10d78d44e84f19d76efba928f65f326093b8e80290e5a794f9c942
Sep 18 14:09:16.609610 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:09:16.609592    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 14:09:17.534704 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:09:17.534673    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bj7wf" event={"ID":"901a17ec-cc37-47c1-bb69-29c51f62e87e","Type":"ContainerStarted","Data":"10723f213e71fd0c73edfa1f263a9bec2e2d084ffb810dc1c2b3767db5b7ac1e"}
Sep 18 14:09:17.535046 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:09:17.534725    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bj7wf" event={"ID":"901a17ec-cc37-47c1-bb69-29c51f62e87e","Type":"ContainerStarted","Data":"256153f3cb10d78d44e84f19d76efba928f65f326093b8e80290e5a794f9c942"}
Sep 18 14:09:17.552743 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:09:17.552690    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bj7wf" podStartSLOduration=1.364261813 podStartE2EDuration="1.552675538s" totalImagesPullingTime="188.413725ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:09:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:09:17.55237705 +0000 UTC m=+629.441093001" watchObservedRunningTime="2026-09-18 14:09:17.552675538 +0000 UTC m=+629.441391467"
Sep 18 14:09:25.590472 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:09:25.590428    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:26.335850 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:09:26.335814    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bj7wf"
Sep 18 14:09:26.336027 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:09:26.335874    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bj7wf"
Sep 18 14:09:26.347929 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:09:26.347907    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bj7wf"
Sep 18 14:09:26.583241 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:09:26.583220    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bj7wf"
Sep 18 14:09:35.036568 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:09:35.036535    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:37.400314 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:09:37.400281    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:51.765275 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:09:51.765231    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:09:51.941818 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:09:51.941787    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:55.120628 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:10:55.120541    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c8788b85-pdmmn"]
Sep 18 14:10:55.125823 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:10:55.125797    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c8788b85-pdmmn"
Sep 18 14:10:55.129373 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:10:55.129350    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 18 14:10:55.129373 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:10:55.129363    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-cbm25\""
Sep 18 14:10:55.133164 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:10:55.133138    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c8788b85-pdmmn"]
Sep 18 14:10:55.226818 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:10:55.226793    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stzzt\" (UniqueName: \"kubernetes.io/projected/a8cc30de-f512-4a5f-90c0-1ff70103cbcd-kube-api-access-stzzt\") pod \"authorino-c8788b85-pdmmn\" (UID: \"a8cc30de-f512-4a5f-90c0-1ff70103cbcd\") " pod="kuadrant-system/authorino-c8788b85-pdmmn"
Sep 18 14:10:55.226934 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:10:55.226823    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a8cc30de-f512-4a5f-90c0-1ff70103cbcd-tls-cert\") pod \"authorino-c8788b85-pdmmn\" (UID: \"a8cc30de-f512-4a5f-90c0-1ff70103cbcd\") " pod="kuadrant-system/authorino-c8788b85-pdmmn"
Sep 18 14:10:55.327710 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:10:55.327681    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stzzt\" (UniqueName: \"kubernetes.io/projected/a8cc30de-f512-4a5f-90c0-1ff70103cbcd-kube-api-access-stzzt\") pod \"authorino-c8788b85-pdmmn\" (UID: \"a8cc30de-f512-4a5f-90c0-1ff70103cbcd\") " pod="kuadrant-system/authorino-c8788b85-pdmmn"
Sep 18 14:10:55.327805 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:10:55.327712    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a8cc30de-f512-4a5f-90c0-1ff70103cbcd-tls-cert\") pod \"authorino-c8788b85-pdmmn\" (UID: \"a8cc30de-f512-4a5f-90c0-1ff70103cbcd\") " pod="kuadrant-system/authorino-c8788b85-pdmmn"
Sep 18 14:10:55.330168 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:10:55.330149    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a8cc30de-f512-4a5f-90c0-1ff70103cbcd-tls-cert\") pod \"authorino-c8788b85-pdmmn\" (UID: \"a8cc30de-f512-4a5f-90c0-1ff70103cbcd\") " pod="kuadrant-system/authorino-c8788b85-pdmmn"
Sep 18 14:10:55.336957 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:10:55.336930    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stzzt\" (UniqueName: \"kubernetes.io/projected/a8cc30de-f512-4a5f-90c0-1ff70103cbcd-kube-api-access-stzzt\") pod \"authorino-c8788b85-pdmmn\" (UID: \"a8cc30de-f512-4a5f-90c0-1ff70103cbcd\") " pod="kuadrant-system/authorino-c8788b85-pdmmn"
Sep 18 14:10:55.436716 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:10:55.436666    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c8788b85-pdmmn"
Sep 18 14:10:55.707839 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:10:55.707769    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c8788b85-pdmmn"]
Sep 18 14:10:55.710691 ip-10-0-143-120 kubenswrapper[2579]: W0918 14:10:55.710668    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda8cc30de_f512_4a5f_90c0_1ff70103cbcd.slice/crio-c7000978a5938a6307a9020abdbcc0374ca6ba0437aff7997cb98ee97826b286 WatchSource:0}: Error finding container c7000978a5938a6307a9020abdbcc0374ca6ba0437aff7997cb98ee97826b286: Status 404 returned error can't find the container with id c7000978a5938a6307a9020abdbcc0374ca6ba0437aff7997cb98ee97826b286
Sep 18 14:10:55.936304 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:10:55.936269    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c8788b85-pdmmn" event={"ID":"a8cc30de-f512-4a5f-90c0-1ff70103cbcd","Type":"ContainerStarted","Data":"c7000978a5938a6307a9020abdbcc0374ca6ba0437aff7997cb98ee97826b286"}
Sep 18 14:10:56.946637 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:10:56.946599    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c8788b85-pdmmn" event={"ID":"a8cc30de-f512-4a5f-90c0-1ff70103cbcd","Type":"ContainerStarted","Data":"422958810c653afa7fc789296650ef4d37140d1d852aa6fcb95b570f51d6e349"}
Sep 18 14:10:56.965407 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:10:56.965359    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c8788b85-pdmmn" podStartSLOduration=0.88342031 podStartE2EDuration="1.965343267s" totalImagesPullingTime="1.081922957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:10:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:10:56.963959903 +0000 UTC m=+728.852675832" watchObservedRunningTime="2026-09-18 14:10:56.965343267 +0000 UTC m=+728.854059197"
Sep 18 14:10:57.571155 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:10:57.571115    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:57.669049 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:10:57.669014    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:58.678357 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:10:58.678318    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:10:58.773420 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:10:58.773385    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:03.937822 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:03.937786    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:08.073934 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:08.073902    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:09.641600 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:09.641563    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:16.035503 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:16.035471    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:16.136778 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:16.136753    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:16.242590 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:16.242565    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:16.391368 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:16.391339    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:16.503498 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:16.503467    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:16.504395 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:16.504373    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh"]
Sep 18 14:11:16.507429 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:16.507413    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh"
Sep 18 14:11:16.510716 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:16.510693    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-dc2cee-openshift-default-dockercfg-5jzw9\""
Sep 18 14:11:16.519049 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:16.519029    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh"]
Sep 18 14:11:16.588954 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:16.588934    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aebd26b0-fc86-42ed-b073-15eddb601b91-istio-data\") pod \"e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh\" (UID: \"aebd26b0-fc86-42ed-b073-15eddb601b91\") " pod="openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh"
Sep 18 14:11:16.589055 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:16.588959    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgwx4\" (UniqueName: \"kubernetes.io/projected/aebd26b0-fc86-42ed-b073-15eddb601b91-kube-api-access-qgwx4\") pod \"e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh\" (UID: \"aebd26b0-fc86-42ed-b073-15eddb601b91\") " pod="openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh"
Sep 18 14:11:16.589055 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:16.588986    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aebd26b0-fc86-42ed-b073-15eddb601b91-workload-socket\") pod \"e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh\" (UID: \"aebd26b0-fc86-42ed-b073-15eddb601b91\") " pod="openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh"
Sep 18 14:11:16.589055 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:16.589006    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aebd26b0-fc86-42ed-b073-15eddb601b91-istio-envoy\") pod \"e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh\" (UID: \"aebd26b0-fc86-42ed-b073-15eddb601b91\") " pod="openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh"
Sep 18 14:11:16.589159 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:16.589086    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aebd26b0-fc86-42ed-b073-15eddb601b91-credential-socket\") pod \"e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh\" (UID: \"aebd26b0-fc86-42ed-b073-15eddb601b91\") " pod="openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh"
Sep 18 14:11:16.589159 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:16.589125    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aebd26b0-fc86-42ed-b073-15eddb601b91-istio-podinfo\") pod \"e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh\" (UID: \"aebd26b0-fc86-42ed-b073-15eddb601b91\") " pod="openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh"
Sep 18 14:11:16.589159 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:16.589142    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aebd26b0-fc86-42ed-b073-15eddb601b91-istio-token\") pod \"e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh\" (UID: \"aebd26b0-fc86-42ed-b073-15eddb601b91\") " pod="openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh"
Sep 18 14:11:16.589264 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:16.589198    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aebd26b0-fc86-42ed-b073-15eddb601b91-workload-certs\") pod \"e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh\" (UID: \"aebd26b0-fc86-42ed-b073-15eddb601b91\") " pod="openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh"
Sep 18 14:11:16.589264 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:16.589214    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aebd26b0-fc86-42ed-b073-15eddb601b91-istiod-ca-cert\") pod \"e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh\" (UID: \"aebd26b0-fc86-42ed-b073-15eddb601b91\") " pod="openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh"
Sep 18 14:11:16.620283 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:16.620256    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:16.689883 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:16.689811    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aebd26b0-fc86-42ed-b073-15eddb601b91-workload-socket\") pod \"e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh\" (UID: \"aebd26b0-fc86-42ed-b073-15eddb601b91\") " pod="openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh"
Sep 18 14:11:16.689883 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:16.689857    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aebd26b0-fc86-42ed-b073-15eddb601b91-istio-envoy\") pod \"e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh\" (UID: \"aebd26b0-fc86-42ed-b073-15eddb601b91\") " pod="openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh"
Sep 18 14:11:16.690044 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:16.689889    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aebd26b0-fc86-42ed-b073-15eddb601b91-credential-socket\") pod \"e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh\" (UID: \"aebd26b0-fc86-42ed-b073-15eddb601b91\") " pod="openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh"
Sep 18 14:11:16.690044 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:16.689921    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aebd26b0-fc86-42ed-b073-15eddb601b91-istio-podinfo\") pod \"e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh\" (UID: \"aebd26b0-fc86-42ed-b073-15eddb601b91\") " pod="openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh"
Sep 18 14:11:16.690044 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:16.689947    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aebd26b0-fc86-42ed-b073-15eddb601b91-istio-token\") pod \"e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh\" (UID: \"aebd26b0-fc86-42ed-b073-15eddb601b91\") " pod="openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh"
Sep 18 14:11:16.690044 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:16.689993    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aebd26b0-fc86-42ed-b073-15eddb601b91-workload-certs\") pod \"e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh\" (UID: \"aebd26b0-fc86-42ed-b073-15eddb601b91\") " pod="openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh"
Sep 18 14:11:16.690044 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:16.690017    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aebd26b0-fc86-42ed-b073-15eddb601b91-istiod-ca-cert\") pod \"e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh\" (UID: \"aebd26b0-fc86-42ed-b073-15eddb601b91\") " pod="openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh"
Sep 18 14:11:16.690334 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:16.690078    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aebd26b0-fc86-42ed-b073-15eddb601b91-istio-data\") pod \"e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh\" (UID: \"aebd26b0-fc86-42ed-b073-15eddb601b91\") " pod="openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh"
Sep 18 14:11:16.690334 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:16.690104    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgwx4\" (UniqueName: \"kubernetes.io/projected/aebd26b0-fc86-42ed-b073-15eddb601b91-kube-api-access-qgwx4\") pod \"e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh\" (UID: \"aebd26b0-fc86-42ed-b073-15eddb601b91\") " pod="openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh"
Sep 18 14:11:16.690334 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:16.690249    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aebd26b0-fc86-42ed-b073-15eddb601b91-workload-socket\") pod \"e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh\" (UID: \"aebd26b0-fc86-42ed-b073-15eddb601b91\") " pod="openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh"
Sep 18 14:11:16.690832 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:16.690792    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aebd26b0-fc86-42ed-b073-15eddb601b91-istio-data\") pod \"e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh\" (UID: \"aebd26b0-fc86-42ed-b073-15eddb601b91\") " pod="openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh"
Sep 18 14:11:16.690943 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:16.690843    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aebd26b0-fc86-42ed-b073-15eddb601b91-credential-socket\") pod \"e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh\" (UID: \"aebd26b0-fc86-42ed-b073-15eddb601b91\") " pod="openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh"
Sep 18 14:11:16.690943 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:16.690888    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aebd26b0-fc86-42ed-b073-15eddb601b91-istiod-ca-cert\") pod \"e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh\" (UID: \"aebd26b0-fc86-42ed-b073-15eddb601b91\") " pod="openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh"
Sep 18 14:11:16.691067 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:16.690971    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aebd26b0-fc86-42ed-b073-15eddb601b91-workload-certs\") pod \"e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh\" (UID: \"aebd26b0-fc86-42ed-b073-15eddb601b91\") " pod="openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh"
Sep 18 14:11:16.692675 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:16.692656    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aebd26b0-fc86-42ed-b073-15eddb601b91-istio-podinfo\") pod \"e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh\" (UID: \"aebd26b0-fc86-42ed-b073-15eddb601b91\") " pod="openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh"
Sep 18 14:11:16.692921 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:16.692904    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aebd26b0-fc86-42ed-b073-15eddb601b91-istio-envoy\") pod \"e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh\" (UID: \"aebd26b0-fc86-42ed-b073-15eddb601b91\") " pod="openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh"
Sep 18 14:11:16.705305 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:16.705280    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aebd26b0-fc86-42ed-b073-15eddb601b91-istio-token\") pod \"e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh\" (UID: \"aebd26b0-fc86-42ed-b073-15eddb601b91\") " pod="openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh"
Sep 18 14:11:16.705727 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:16.705705    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgwx4\" (UniqueName: \"kubernetes.io/projected/aebd26b0-fc86-42ed-b073-15eddb601b91-kube-api-access-qgwx4\") pod \"e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh\" (UID: \"aebd26b0-fc86-42ed-b073-15eddb601b91\") " pod="openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh"
Sep 18 14:11:16.744243 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:16.744225    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:16.818384 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:16.818368    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh"
Sep 18 14:11:17.006864 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:17.006791    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:17.095384 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:17.095362    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh"]
Sep 18 14:11:17.097218 ip-10-0-143-120 kubenswrapper[2579]: W0918 14:11:17.097192    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaebd26b0_fc86_42ed_b073_15eddb601b91.slice/crio-21280a457ef4b76582f97b5b44937d52f7afe05487afc135cab3505e952519f3 WatchSource:0}: Error finding container 21280a457ef4b76582f97b5b44937d52f7afe05487afc135cab3505e952519f3: Status 404 returned error can't find the container with id 21280a457ef4b76582f97b5b44937d52f7afe05487afc135cab3505e952519f3
Sep 18 14:11:17.484289 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:17.484252    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:17.582547 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:17.582514    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:17.676430 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:17.676396    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:17.909488 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:17.909455    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:18.038965 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:18.038928    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh" event={"ID":"aebd26b0-fc86-42ed-b073-15eddb601b91","Type":"ContainerStarted","Data":"21280a457ef4b76582f97b5b44937d52f7afe05487afc135cab3505e952519f3"}
Sep 18 14:11:19.565799 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:19.565766    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:19.684518 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:19.681928    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:19.827137 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:19.827100    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:11:19.827240 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:19.827185    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:11:19.827240 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:19.827219    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:11:19.842923 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:19.842899    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:20.049280 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:20.049237    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh" event={"ID":"aebd26b0-fc86-42ed-b073-15eddb601b91","Type":"ContainerStarted","Data":"a4828280ba2425af30941e863a455d11397546946b1dfd655ff686b2fa931ba4"}
Sep 18 14:11:20.084214 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:20.084113    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh" podStartSLOduration=1.356308439 podStartE2EDuration="4.084096939s" totalImagesPullingTime="2.7277885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:11:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:11:20.082994904 +0000 UTC m=+751.971710834" watchObservedRunningTime="2026-09-18 14:11:20.084096939 +0000 UTC m=+751.972812943"
Sep 18 14:11:20.818687 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:20.818654    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh"
Sep 18 14:11:20.826256 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:20.826225    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh"
Sep 18 14:11:21.054669 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:21.054634    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh"
Sep 18 14:11:21.055898 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:21.055872    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh"
Sep 18 14:11:21.206674 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:21.206606    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-4d9222-7d5bb89d47-c2shv"]
Sep 18 14:11:21.209513 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:21.209492    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4d9222-7d5bb89d47-c2shv"
Sep 18 14:11:21.214538 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:21.214516    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-4d9222-serving-cert\""
Sep 18 14:11:21.230411 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:21.230383    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-4d9222-7d5bb89d47-c2shv"]
Sep 18 14:11:21.264964 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:21.264938    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:21.333829 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:21.333806    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/938f2892-9feb-4658-acc8-5303503bcf05-maas-api-tls\") pod \"maas-api-e2e-worker-w0-4d9222-7d5bb89d47-c2shv\" (UID: \"938f2892-9feb-4658-acc8-5303503bcf05\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4d9222-7d5bb89d47-c2shv"
Sep 18 14:11:21.333938 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:21.333864    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8m87\" (UniqueName: \"kubernetes.io/projected/938f2892-9feb-4658-acc8-5303503bcf05-kube-api-access-s8m87\") pod \"maas-api-e2e-worker-w0-4d9222-7d5bb89d47-c2shv\" (UID: \"938f2892-9feb-4658-acc8-5303503bcf05\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4d9222-7d5bb89d47-c2shv"
Sep 18 14:11:21.333938 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:21.333884    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/938f2892-9feb-4658-acc8-5303503bcf05-metrics-certs\") pod \"maas-api-e2e-worker-w0-4d9222-7d5bb89d47-c2shv\" (UID: \"938f2892-9feb-4658-acc8-5303503bcf05\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4d9222-7d5bb89d47-c2shv"
Sep 18 14:11:21.333938 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:21.333903    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/938f2892-9feb-4658-acc8-5303503bcf05-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-4d9222-7d5bb89d47-c2shv\" (UID: \"938f2892-9feb-4658-acc8-5303503bcf05\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4d9222-7d5bb89d47-c2shv"
Sep 18 14:11:21.373415 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:21.373390    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:21.434918 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:21.434892    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8m87\" (UniqueName: \"kubernetes.io/projected/938f2892-9feb-4658-acc8-5303503bcf05-kube-api-access-s8m87\") pod \"maas-api-e2e-worker-w0-4d9222-7d5bb89d47-c2shv\" (UID: \"938f2892-9feb-4658-acc8-5303503bcf05\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4d9222-7d5bb89d47-c2shv"
Sep 18 14:11:21.434918 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:21.434920    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/938f2892-9feb-4658-acc8-5303503bcf05-metrics-certs\") pod \"maas-api-e2e-worker-w0-4d9222-7d5bb89d47-c2shv\" (UID: \"938f2892-9feb-4658-acc8-5303503bcf05\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4d9222-7d5bb89d47-c2shv"
Sep 18 14:11:21.435097 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:21.434942    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/938f2892-9feb-4658-acc8-5303503bcf05-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-4d9222-7d5bb89d47-c2shv\" (UID: \"938f2892-9feb-4658-acc8-5303503bcf05\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4d9222-7d5bb89d47-c2shv"
Sep 18 14:11:21.435097 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:21.435013    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/938f2892-9feb-4658-acc8-5303503bcf05-maas-api-tls\") pod \"maas-api-e2e-worker-w0-4d9222-7d5bb89d47-c2shv\" (UID: \"938f2892-9feb-4658-acc8-5303503bcf05\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4d9222-7d5bb89d47-c2shv"
Sep 18 14:11:21.435220 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:11:21.435119    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-4d9222-serving-cert: secret "maas-api-e2e-worker-w0-4d9222-serving-cert" not found
Sep 18 14:11:21.435220 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:11:21.435208    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/938f2892-9feb-4658-acc8-5303503bcf05-maas-api-tls podName:938f2892-9feb-4658-acc8-5303503bcf05 nodeName:}" failed. No retries permitted until 2026-09-18 14:11:21.935162829 +0000 UTC m=+753.823878737 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/938f2892-9feb-4658-acc8-5303503bcf05-maas-api-tls") pod "maas-api-e2e-worker-w0-4d9222-7d5bb89d47-c2shv" (UID: "938f2892-9feb-4658-acc8-5303503bcf05") : secret "maas-api-e2e-worker-w0-4d9222-serving-cert" not found
Sep 18 14:11:21.435749 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:21.435724    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/938f2892-9feb-4658-acc8-5303503bcf05-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-4d9222-7d5bb89d47-c2shv\" (UID: \"938f2892-9feb-4658-acc8-5303503bcf05\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4d9222-7d5bb89d47-c2shv"
Sep 18 14:11:21.437351 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:21.437333    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/938f2892-9feb-4658-acc8-5303503bcf05-metrics-certs\") pod \"maas-api-e2e-worker-w0-4d9222-7d5bb89d47-c2shv\" (UID: \"938f2892-9feb-4658-acc8-5303503bcf05\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4d9222-7d5bb89d47-c2shv"
Sep 18 14:11:21.451664 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:21.451642    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8m87\" (UniqueName: \"kubernetes.io/projected/938f2892-9feb-4658-acc8-5303503bcf05-kube-api-access-s8m87\") pod \"maas-api-e2e-worker-w0-4d9222-7d5bb89d47-c2shv\" (UID: \"938f2892-9feb-4658-acc8-5303503bcf05\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4d9222-7d5bb89d47-c2shv"
Sep 18 14:11:21.723819 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:21.723797    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:21.938037 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:21.938008    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/938f2892-9feb-4658-acc8-5303503bcf05-maas-api-tls\") pod \"maas-api-e2e-worker-w0-4d9222-7d5bb89d47-c2shv\" (UID: \"938f2892-9feb-4658-acc8-5303503bcf05\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4d9222-7d5bb89d47-c2shv"
Sep 18 14:11:21.940433 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:21.940413    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/938f2892-9feb-4658-acc8-5303503bcf05-maas-api-tls\") pod \"maas-api-e2e-worker-w0-4d9222-7d5bb89d47-c2shv\" (UID: \"938f2892-9feb-4658-acc8-5303503bcf05\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4d9222-7d5bb89d47-c2shv"
Sep 18 14:11:22.082397 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:22.082368    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:22.121284 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:22.121253    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4d9222-7d5bb89d47-c2shv"
Sep 18 14:11:22.186957 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:22.186778    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:22.439485 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:22.439411    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:22.446647 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:22.446621    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-4d9222-7d5bb89d47-c2shv"]
Sep 18 14:11:22.450054 ip-10-0-143-120 kubenswrapper[2579]: W0918 14:11:22.450028    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod938f2892_9feb_4658_acc8_5303503bcf05.slice/crio-ee439c8c240fe3e01d394afbd64a425006a9ccff02f2d4ed148551feb5283db0 WatchSource:0}: Error finding container ee439c8c240fe3e01d394afbd64a425006a9ccff02f2d4ed148551feb5283db0: Status 404 returned error can't find the container with id ee439c8c240fe3e01d394afbd64a425006a9ccff02f2d4ed148551feb5283db0
Sep 18 14:11:22.596480 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:22.596453    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:22.895585 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:22.895562    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-93827d-cdb46b949-4rwcm"]
Sep 18 14:11:22.900078 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:22.900058    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-93827d-cdb46b949-4rwcm"
Sep 18 14:11:22.902722 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:22.902700    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-93827d-serving-cert\""
Sep 18 14:11:22.922720 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:22.922693    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-93827d-cdb46b949-4rwcm"]
Sep 18 14:11:22.961701 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:22.961683    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:23.046031 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:23.046002    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d516717-4809-4020-af5b-3cb8478f112d-maas-api-tls\") pod \"maas-api-e2e-worker-w4-93827d-cdb46b949-4rwcm\" (UID: \"0d516717-4809-4020-af5b-3cb8478f112d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-93827d-cdb46b949-4rwcm"
Sep 18 14:11:23.046164 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:23.046040    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d516717-4809-4020-af5b-3cb8478f112d-metrics-certs\") pod \"maas-api-e2e-worker-w4-93827d-cdb46b949-4rwcm\" (UID: \"0d516717-4809-4020-af5b-3cb8478f112d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-93827d-cdb46b949-4rwcm"
Sep 18 14:11:23.046164 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:23.046067    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d516717-4809-4020-af5b-3cb8478f112d-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-93827d-cdb46b949-4rwcm\" (UID: \"0d516717-4809-4020-af5b-3cb8478f112d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-93827d-cdb46b949-4rwcm"
Sep 18 14:11:23.046298 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:23.046216    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms6sl\" (UniqueName: \"kubernetes.io/projected/0d516717-4809-4020-af5b-3cb8478f112d-kube-api-access-ms6sl\") pod \"maas-api-e2e-worker-w4-93827d-cdb46b949-4rwcm\" (UID: \"0d516717-4809-4020-af5b-3cb8478f112d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-93827d-cdb46b949-4rwcm"
Sep 18 14:11:23.061547 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:23.061522    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:23.064966 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:23.064941    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4d9222-7d5bb89d47-c2shv" event={"ID":"938f2892-9feb-4658-acc8-5303503bcf05","Type":"ContainerStarted","Data":"152f68837325a11e8fae70818ee11db240d3199b10a188116da4027fcf2dc893"}
Sep 18 14:11:23.065081 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:23.064987    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4d9222-7d5bb89d47-c2shv" event={"ID":"938f2892-9feb-4658-acc8-5303503bcf05","Type":"ContainerStarted","Data":"ee439c8c240fe3e01d394afbd64a425006a9ccff02f2d4ed148551feb5283db0"}
Sep 18 14:11:23.065163 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:23.065148    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4d9222-7d5bb89d47-c2shv"
Sep 18 14:11:23.109729 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:23.109686    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4d9222-7d5bb89d47-c2shv" podStartSLOduration=1.676614239 podStartE2EDuration="2.10967267s" totalImagesPullingTime="433.058431ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:11:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:11:23.108314926 +0000 UTC m=+754.997030858" watchObservedRunningTime="2026-09-18 14:11:23.10967267 +0000 UTC m=+754.998388600"
Sep 18 14:11:23.146764 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:23.146706    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d516717-4809-4020-af5b-3cb8478f112d-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-93827d-cdb46b949-4rwcm\" (UID: \"0d516717-4809-4020-af5b-3cb8478f112d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-93827d-cdb46b949-4rwcm"
Sep 18 14:11:23.146868 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:23.146856    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms6sl\" (UniqueName: \"kubernetes.io/projected/0d516717-4809-4020-af5b-3cb8478f112d-kube-api-access-ms6sl\") pod \"maas-api-e2e-worker-w4-93827d-cdb46b949-4rwcm\" (UID: \"0d516717-4809-4020-af5b-3cb8478f112d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-93827d-cdb46b949-4rwcm"
Sep 18 14:11:23.147306 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:23.147124    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d516717-4809-4020-af5b-3cb8478f112d-maas-api-tls\") pod \"maas-api-e2e-worker-w4-93827d-cdb46b949-4rwcm\" (UID: \"0d516717-4809-4020-af5b-3cb8478f112d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-93827d-cdb46b949-4rwcm"
Sep 18 14:11:23.147306 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:23.147215    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d516717-4809-4020-af5b-3cb8478f112d-metrics-certs\") pod \"maas-api-e2e-worker-w4-93827d-cdb46b949-4rwcm\" (UID: \"0d516717-4809-4020-af5b-3cb8478f112d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-93827d-cdb46b949-4rwcm"
Sep 18 14:11:23.147499 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:23.147479    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d516717-4809-4020-af5b-3cb8478f112d-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-93827d-cdb46b949-4rwcm\" (UID: \"0d516717-4809-4020-af5b-3cb8478f112d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-93827d-cdb46b949-4rwcm"
Sep 18 14:11:23.149613 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:23.149584    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d516717-4809-4020-af5b-3cb8478f112d-metrics-certs\") pod \"maas-api-e2e-worker-w4-93827d-cdb46b949-4rwcm\" (UID: \"0d516717-4809-4020-af5b-3cb8478f112d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-93827d-cdb46b949-4rwcm"
Sep 18 14:11:23.149765 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:23.149733    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d516717-4809-4020-af5b-3cb8478f112d-maas-api-tls\") pod \"maas-api-e2e-worker-w4-93827d-cdb46b949-4rwcm\" (UID: \"0d516717-4809-4020-af5b-3cb8478f112d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-93827d-cdb46b949-4rwcm"
Sep 18 14:11:23.155951 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:23.155929    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms6sl\" (UniqueName: \"kubernetes.io/projected/0d516717-4809-4020-af5b-3cb8478f112d-kube-api-access-ms6sl\") pod \"maas-api-e2e-worker-w4-93827d-cdb46b949-4rwcm\" (UID: \"0d516717-4809-4020-af5b-3cb8478f112d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-93827d-cdb46b949-4rwcm"
Sep 18 14:11:23.164334 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:23.164314    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:23.237340 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:23.237314    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-93827d-cdb46b949-4rwcm"
Sep 18 14:11:23.521466 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:23.521439    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-93827d-cdb46b949-4rwcm"]
Sep 18 14:11:23.521810 ip-10-0-143-120 kubenswrapper[2579]: W0918 14:11:23.521788    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d516717_4809_4020_af5b_3cb8478f112d.slice/crio-a0b0bca4643d6697800d55556dbadff60b593c20229b9dacf922764be8a230dd WatchSource:0}: Error finding container a0b0bca4643d6697800d55556dbadff60b593c20229b9dacf922764be8a230dd: Status 404 returned error can't find the container with id a0b0bca4643d6697800d55556dbadff60b593c20229b9dacf922764be8a230dd
Sep 18 14:11:23.552345 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:23.552311    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:23.641326 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:23.641301    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:24.022637 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:24.022606    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:24.073407 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:24.073079    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-93827d-cdb46b949-4rwcm" event={"ID":"0d516717-4809-4020-af5b-3cb8478f112d","Type":"ContainerStarted","Data":"013ce6a9dcbce1c2636301d63fcdc205e28a74037a2f59255af46dbba9de9bff"}
Sep 18 14:11:24.073407 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:24.073130    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-93827d-cdb46b949-4rwcm" event={"ID":"0d516717-4809-4020-af5b-3cb8478f112d","Type":"ContainerStarted","Data":"a0b0bca4643d6697800d55556dbadff60b593c20229b9dacf922764be8a230dd"}
Sep 18 14:11:24.073407 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:24.073194    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-93827d-cdb46b949-4rwcm"
Sep 18 14:11:24.099448 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:24.099393    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-93827d-cdb46b949-4rwcm" podStartSLOduration=1.764341726 podStartE2EDuration="2.099376454s" totalImagesPullingTime="335.034728ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:11:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:11:24.098235047 +0000 UTC m=+755.986950977" watchObservedRunningTime="2026-09-18 14:11:24.099376454 +0000 UTC m=+755.988092385"
Sep 18 14:11:24.551792 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:24.551758    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:24.888183 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:24.888097    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:25.222699 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:25.222616    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2cee-546c4854cc-mvws4"]
Sep 18 14:11:25.226101 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:25.226079    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2cee-546c4854cc-mvws4"
Sep 18 14:11:25.228706 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:25.228682    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-dc2cee-serving-cert\""
Sep 18 14:11:25.245386 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:25.245361    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2cee-546c4854cc-mvws4"]
Sep 18 14:11:25.278222 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:25.278189    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:25.370565 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:25.370537    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2669c0c-d001-41b0-be3c-42cb419ab807-openshift-service-ca\") pod \"maas-api-e2e-inf-a-dc2cee-546c4854cc-mvws4\" (UID: \"f2669c0c-d001-41b0-be3c-42cb419ab807\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2cee-546c4854cc-mvws4"
Sep 18 14:11:25.370715 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:25.370584    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2669c0c-d001-41b0-be3c-42cb419ab807-maas-api-tls\") pod \"maas-api-e2e-inf-a-dc2cee-546c4854cc-mvws4\" (UID: \"f2669c0c-d001-41b0-be3c-42cb419ab807\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2cee-546c4854cc-mvws4"
Sep 18 14:11:25.370715 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:25.370649    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2669c0c-d001-41b0-be3c-42cb419ab807-metrics-certs\") pod \"maas-api-e2e-inf-a-dc2cee-546c4854cc-mvws4\" (UID: \"f2669c0c-d001-41b0-be3c-42cb419ab807\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2cee-546c4854cc-mvws4"
Sep 18 14:11:25.370715 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:25.370693    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqtf5\" (UniqueName: \"kubernetes.io/projected/f2669c0c-d001-41b0-be3c-42cb419ab807-kube-api-access-tqtf5\") pod \"maas-api-e2e-inf-a-dc2cee-546c4854cc-mvws4\" (UID: \"f2669c0c-d001-41b0-be3c-42cb419ab807\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2cee-546c4854cc-mvws4"
Sep 18 14:11:25.384057 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:25.384029    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:25.476293 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:25.476232    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2669c0c-d001-41b0-be3c-42cb419ab807-openshift-service-ca\") pod \"maas-api-e2e-inf-a-dc2cee-546c4854cc-mvws4\" (UID: \"f2669c0c-d001-41b0-be3c-42cb419ab807\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2cee-546c4854cc-mvws4"
Sep 18 14:11:25.476293 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:25.476271    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2669c0c-d001-41b0-be3c-42cb419ab807-maas-api-tls\") pod \"maas-api-e2e-inf-a-dc2cee-546c4854cc-mvws4\" (UID: \"f2669c0c-d001-41b0-be3c-42cb419ab807\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2cee-546c4854cc-mvws4"
Sep 18 14:11:25.476425 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:25.476300    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2669c0c-d001-41b0-be3c-42cb419ab807-metrics-certs\") pod \"maas-api-e2e-inf-a-dc2cee-546c4854cc-mvws4\" (UID: \"f2669c0c-d001-41b0-be3c-42cb419ab807\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2cee-546c4854cc-mvws4"
Sep 18 14:11:25.476425 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:25.476324    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqtf5\" (UniqueName: \"kubernetes.io/projected/f2669c0c-d001-41b0-be3c-42cb419ab807-kube-api-access-tqtf5\") pod \"maas-api-e2e-inf-a-dc2cee-546c4854cc-mvws4\" (UID: \"f2669c0c-d001-41b0-be3c-42cb419ab807\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2cee-546c4854cc-mvws4"
Sep 18 14:11:25.476425 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:11:25.476381    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2cee-serving-cert: secret "maas-api-e2e-inf-a-dc2cee-serving-cert" not found
Sep 18 14:11:25.476537 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:11:25.476479    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2669c0c-d001-41b0-be3c-42cb419ab807-maas-api-tls podName:f2669c0c-d001-41b0-be3c-42cb419ab807 nodeName:}" failed. No retries permitted until 2026-09-18 14:11:25.976459316 +0000 UTC m=+757.865175234 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f2669c0c-d001-41b0-be3c-42cb419ab807-maas-api-tls") pod "maas-api-e2e-inf-a-dc2cee-546c4854cc-mvws4" (UID: "f2669c0c-d001-41b0-be3c-42cb419ab807") : secret "maas-api-e2e-inf-a-dc2cee-serving-cert" not found
Sep 18 14:11:25.476935 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:25.476916    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2669c0c-d001-41b0-be3c-42cb419ab807-openshift-service-ca\") pod \"maas-api-e2e-inf-a-dc2cee-546c4854cc-mvws4\" (UID: \"f2669c0c-d001-41b0-be3c-42cb419ab807\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2cee-546c4854cc-mvws4"
Sep 18 14:11:25.478832 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:25.478810    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2669c0c-d001-41b0-be3c-42cb419ab807-metrics-certs\") pod \"maas-api-e2e-inf-a-dc2cee-546c4854cc-mvws4\" (UID: \"f2669c0c-d001-41b0-be3c-42cb419ab807\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2cee-546c4854cc-mvws4"
Sep 18 14:11:25.485580 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:25.485562    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqtf5\" (UniqueName: \"kubernetes.io/projected/f2669c0c-d001-41b0-be3c-42cb419ab807-kube-api-access-tqtf5\") pod \"maas-api-e2e-inf-a-dc2cee-546c4854cc-mvws4\" (UID: \"f2669c0c-d001-41b0-be3c-42cb419ab807\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2cee-546c4854cc-mvws4"
Sep 18 14:11:25.566895 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:25.566848    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:25.981390 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:25.981361    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2669c0c-d001-41b0-be3c-42cb419ab807-maas-api-tls\") pod \"maas-api-e2e-inf-a-dc2cee-546c4854cc-mvws4\" (UID: \"f2669c0c-d001-41b0-be3c-42cb419ab807\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2cee-546c4854cc-mvws4"
Sep 18 14:11:25.983776 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:25.983756    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2669c0c-d001-41b0-be3c-42cb419ab807-maas-api-tls\") pod \"maas-api-e2e-inf-a-dc2cee-546c4854cc-mvws4\" (UID: \"f2669c0c-d001-41b0-be3c-42cb419ab807\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2cee-546c4854cc-mvws4"
Sep 18 14:11:26.027227 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:26.027198    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:26.136217 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:26.136186    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2cee-546c4854cc-mvws4"
Sep 18 14:11:26.162944 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:26.162916    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:26.434519 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:26.434496    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2cee-546c4854cc-mvws4"]
Sep 18 14:11:26.435615 ip-10-0-143-120 kubenswrapper[2579]: W0918 14:11:26.435593    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2669c0c_d001_41b0_be3c_42cb419ab807.slice/crio-e5f70a8b3d4603b04d57235bf159f48b785b8ccbf6044cc18c725640f97d6a8f WatchSource:0}: Error finding container e5f70a8b3d4603b04d57235bf159f48b785b8ccbf6044cc18c725640f97d6a8f: Status 404 returned error can't find the container with id e5f70a8b3d4603b04d57235bf159f48b785b8ccbf6044cc18c725640f97d6a8f
Sep 18 14:11:26.453067 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:26.453048    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:26.607644 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:26.607617    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:26.769285 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:26.769219    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:26.986373 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:26.986328    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:27.077121 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:27.077092    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m"]
Sep 18 14:11:27.079814 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:27.079798    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m"
Sep 18 14:11:27.082358 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:27.082336    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-eeb0a3f4-openshift-default-dockercfg-qrh2g\""
Sep 18 14:11:27.085832 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:27.085802    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2cee-546c4854cc-mvws4" event={"ID":"f2669c0c-d001-41b0-be3c-42cb419ab807","Type":"ContainerStarted","Data":"c4be97e937a24c275a12dfc3706dabcb39f3cb41ed582287eea5daa2de772677"}
Sep 18 14:11:27.085964 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:27.085850    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2cee-546c4854cc-mvws4" event={"ID":"f2669c0c-d001-41b0-be3c-42cb419ab807","Type":"ContainerStarted","Data":"e5f70a8b3d4603b04d57235bf159f48b785b8ccbf6044cc18c725640f97d6a8f"}
Sep 18 14:11:27.085964 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:27.085952    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2cee-546c4854cc-mvws4"
Sep 18 14:11:27.156899 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:27.156874    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:27.167135 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:27.167076    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m"]
Sep 18 14:11:27.169063 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:27.169017    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2cee-546c4854cc-mvws4" podStartSLOduration=1.8032445 podStartE2EDuration="2.168998171s" totalImagesPullingTime="365.753671ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:11:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:11:27.159710189 +0000 UTC m=+759.048426120" watchObservedRunningTime="2026-09-18 14:11:27.168998171 +0000 UTC m=+759.057714103"
Sep 18 14:11:27.191473 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:27.191102    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-istio-podinfo\") pod \"e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m\" (UID: \"1cefcc6a-27d6-4530-821e-a3c1f0272cd8\") " pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m"
Sep 18 14:11:27.191621 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:27.191502    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-credential-socket\") pod \"e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m\" (UID: \"1cefcc6a-27d6-4530-821e-a3c1f0272cd8\") " pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m"
Sep 18 14:11:27.191621 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:27.191538    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-istio-data\") pod \"e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m\" (UID: \"1cefcc6a-27d6-4530-821e-a3c1f0272cd8\") " pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m"
Sep 18 14:11:27.191621 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:27.191604    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwvhl\" (UniqueName: \"kubernetes.io/projected/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-kube-api-access-lwvhl\") pod \"e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m\" (UID: \"1cefcc6a-27d6-4530-821e-a3c1f0272cd8\") " pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m"
Sep 18 14:11:27.191876 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:27.191845    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-istiod-ca-cert\") pod \"e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m\" (UID: \"1cefcc6a-27d6-4530-821e-a3c1f0272cd8\") " pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m"
Sep 18 14:11:27.192188 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:27.191943    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-istio-token\") pod \"e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m\" (UID: \"1cefcc6a-27d6-4530-821e-a3c1f0272cd8\") " pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m"
Sep 18 14:11:27.192188 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:27.191972    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-istio-envoy\") pod \"e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m\" (UID: \"1cefcc6a-27d6-4530-821e-a3c1f0272cd8\") " pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m"
Sep 18 14:11:27.192470 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:27.192448    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-workload-certs\") pod \"e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m\" (UID: \"1cefcc6a-27d6-4530-821e-a3c1f0272cd8\") " pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m"
Sep 18 14:11:27.192546 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:27.192506    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-workload-socket\") pod \"e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m\" (UID: \"1cefcc6a-27d6-4530-821e-a3c1f0272cd8\") " pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m"
Sep 18 14:11:27.293683 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:27.293641    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-credential-socket\") pod \"e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m\" (UID: \"1cefcc6a-27d6-4530-821e-a3c1f0272cd8\") " pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m"
Sep 18 14:11:27.294429 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:27.294271    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-credential-socket\") pod \"e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m\" (UID: \"1cefcc6a-27d6-4530-821e-a3c1f0272cd8\") " pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m"
Sep 18 14:11:27.294571 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:27.294368    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-istio-data\") pod \"e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m\" (UID: \"1cefcc6a-27d6-4530-821e-a3c1f0272cd8\") " pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m"
Sep 18 14:11:27.294658 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:27.294634    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwvhl\" (UniqueName: \"kubernetes.io/projected/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-kube-api-access-lwvhl\") pod \"e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m\" (UID: \"1cefcc6a-27d6-4530-821e-a3c1f0272cd8\") " pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m"
Sep 18 14:11:27.294785 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:27.294681    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-istiod-ca-cert\") pod \"e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m\" (UID: \"1cefcc6a-27d6-4530-821e-a3c1f0272cd8\") " pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m"
Sep 18 14:11:27.294785 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:27.294721    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-istio-token\") pod \"e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m\" (UID: \"1cefcc6a-27d6-4530-821e-a3c1f0272cd8\") " pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m"
Sep 18 14:11:27.294785 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:27.294753    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-istio-envoy\") pod \"e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m\" (UID: \"1cefcc6a-27d6-4530-821e-a3c1f0272cd8\") " pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m"
Sep 18 14:11:27.294936 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:27.294831    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-workload-certs\") pod \"e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m\" (UID: \"1cefcc6a-27d6-4530-821e-a3c1f0272cd8\") " pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m"
Sep 18 14:11:27.294936 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:27.294868    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-workload-socket\") pod \"e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m\" (UID: \"1cefcc6a-27d6-4530-821e-a3c1f0272cd8\") " pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m"
Sep 18 14:11:27.294936 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:27.294879    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-istio-data\") pod \"e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m\" (UID: \"1cefcc6a-27d6-4530-821e-a3c1f0272cd8\") " pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m"
Sep 18 14:11:27.295079 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:27.294950    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-istio-podinfo\") pod \"e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m\" (UID: \"1cefcc6a-27d6-4530-821e-a3c1f0272cd8\") " pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m"
Sep 18 14:11:27.296143 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:27.296095    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-istiod-ca-cert\") pod \"e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m\" (UID: \"1cefcc6a-27d6-4530-821e-a3c1f0272cd8\") " pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m"
Sep 18 14:11:27.296573 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:27.296548    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-workload-socket\") pod \"e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m\" (UID: \"1cefcc6a-27d6-4530-821e-a3c1f0272cd8\") " pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m"
Sep 18 14:11:27.296835 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:27.296812    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-workload-certs\") pod \"e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m\" (UID: \"1cefcc6a-27d6-4530-821e-a3c1f0272cd8\") " pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m"
Sep 18 14:11:27.298153 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:27.298131    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-istio-podinfo\") pod \"e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m\" (UID: \"1cefcc6a-27d6-4530-821e-a3c1f0272cd8\") " pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m"
Sep 18 14:11:27.298695 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:27.298673    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-istio-envoy\") pod \"e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m\" (UID: \"1cefcc6a-27d6-4530-821e-a3c1f0272cd8\") " pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m"
Sep 18 14:11:27.304064 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:27.304036    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-istio-token\") pod \"e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m\" (UID: \"1cefcc6a-27d6-4530-821e-a3c1f0272cd8\") " pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m"
Sep 18 14:11:27.304535 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:27.304508    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwvhl\" (UniqueName: \"kubernetes.io/projected/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-kube-api-access-lwvhl\") pod \"e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m\" (UID: \"1cefcc6a-27d6-4530-821e-a3c1f0272cd8\") " pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m"
Sep 18 14:11:27.390057 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:27.389998    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m"
Sep 18 14:11:27.435462 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:27.435404    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:27.722011 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:27.721978    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m"]
Sep 18 14:11:27.725195 ip-10-0-143-120 kubenswrapper[2579]: W0918 14:11:27.725145    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cefcc6a_27d6_4530_821e_a3c1f0272cd8.slice/crio-79cd28e65b8a7e12a1257cad0888ea543b307b26c78d5c322fc82bec21e1b374 WatchSource:0}: Error finding container 79cd28e65b8a7e12a1257cad0888ea543b307b26c78d5c322fc82bec21e1b374: Status 404 returned error can't find the container with id 79cd28e65b8a7e12a1257cad0888ea543b307b26c78d5c322fc82bec21e1b374
Sep 18 14:11:27.727597 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:27.727556    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:11:27.727706 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:27.727637    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:11:27.727706 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:27.727681    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:11:28.091485 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:28.091450    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m" event={"ID":"1cefcc6a-27d6-4530-821e-a3c1f0272cd8","Type":"ContainerStarted","Data":"9e51caa5b3fb538fa2377e893fae057d1d09cc648b31165692b33dce1f3133c4"}
Sep 18 14:11:28.091632 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:28.091513    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m" event={"ID":"1cefcc6a-27d6-4530-821e-a3c1f0272cd8","Type":"ContainerStarted","Data":"79cd28e65b8a7e12a1257cad0888ea543b307b26c78d5c322fc82bec21e1b374"}
Sep 18 14:11:28.136420 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:28.136362    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m" podStartSLOduration=1.136340967 podStartE2EDuration="1.136340967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:11:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:11:28.135645247 +0000 UTC m=+760.024361177" watchObservedRunningTime="2026-09-18 14:11:28.136340967 +0000 UTC m=+760.025056900"
Sep 18 14:11:28.390498 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:28.390416    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m"
Sep 18 14:11:28.396862 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:28.396833    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m"
Sep 18 14:11:28.619950 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:28.619914    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:29.078716 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:29.078689    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4d9222-7d5bb89d47-c2shv"
Sep 18 14:11:29.095872 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:29.095837    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m"
Sep 18 14:11:29.096992 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:29.096973    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m"
Sep 18 14:11:29.697560 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:29.697529    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:29.807089 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:29.807065    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:30.281210 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:30.281161    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:30.579098 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:30.579072    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:30.702127 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:30.702101    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:31.085093 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:31.085070    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-93827d-cdb46b949-4rwcm"
Sep 18 14:11:31.980250 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:31.980217    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:32.045336 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:32.045305    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:32.332410 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:32.332375    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:32.777429 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:32.777357    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:32.878590 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:32.878558    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:33.096716 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:33.096690    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2cee-546c4854cc-mvws4"
Sep 18 14:11:33.503216 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:33.503126    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:34.599708 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:34.599672    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:35.503658 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:35.503623    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-ae7021-ccb785d8c-ztcqg"]
Sep 18 14:11:35.507274 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:35.507249    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ae7021-ccb785d8c-ztcqg"
Sep 18 14:11:35.510123 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:35.510103    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-ae7021-serving-cert\""
Sep 18 14:11:35.518016 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:35.517993    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-ae7021-ccb785d8c-ztcqg"]
Sep 18 14:11:35.546352 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:35.546326    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:35.569709 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:35.569685    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bee7612-56a4-4f90-a62a-8a7674f5634d-maas-api-tls\") pod \"maas-api-e2e-autores-ae7021-ccb785d8c-ztcqg\" (UID: \"0bee7612-56a4-4f90-a62a-8a7674f5634d\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ae7021-ccb785d8c-ztcqg"
Sep 18 14:11:35.569809 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:35.569723    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bee7612-56a4-4f90-a62a-8a7674f5634d-openshift-service-ca\") pod \"maas-api-e2e-autores-ae7021-ccb785d8c-ztcqg\" (UID: \"0bee7612-56a4-4f90-a62a-8a7674f5634d\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ae7021-ccb785d8c-ztcqg"
Sep 18 14:11:35.569809 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:35.569797    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bee7612-56a4-4f90-a62a-8a7674f5634d-metrics-certs\") pod \"maas-api-e2e-autores-ae7021-ccb785d8c-ztcqg\" (UID: \"0bee7612-56a4-4f90-a62a-8a7674f5634d\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ae7021-ccb785d8c-ztcqg"
Sep 18 14:11:35.569919 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:35.569831    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z6ns\" (UniqueName: \"kubernetes.io/projected/0bee7612-56a4-4f90-a62a-8a7674f5634d-kube-api-access-4z6ns\") pod \"maas-api-e2e-autores-ae7021-ccb785d8c-ztcqg\" (UID: \"0bee7612-56a4-4f90-a62a-8a7674f5634d\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ae7021-ccb785d8c-ztcqg"
Sep 18 14:11:35.645230 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:35.645195    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:35.670671 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:35.670634    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bee7612-56a4-4f90-a62a-8a7674f5634d-openshift-service-ca\") pod \"maas-api-e2e-autores-ae7021-ccb785d8c-ztcqg\" (UID: \"0bee7612-56a4-4f90-a62a-8a7674f5634d\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ae7021-ccb785d8c-ztcqg"
Sep 18 14:11:35.670805 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:35.670695    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bee7612-56a4-4f90-a62a-8a7674f5634d-metrics-certs\") pod \"maas-api-e2e-autores-ae7021-ccb785d8c-ztcqg\" (UID: \"0bee7612-56a4-4f90-a62a-8a7674f5634d\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ae7021-ccb785d8c-ztcqg"
Sep 18 14:11:35.670805 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:35.670712    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z6ns\" (UniqueName: \"kubernetes.io/projected/0bee7612-56a4-4f90-a62a-8a7674f5634d-kube-api-access-4z6ns\") pod \"maas-api-e2e-autores-ae7021-ccb785d8c-ztcqg\" (UID: \"0bee7612-56a4-4f90-a62a-8a7674f5634d\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ae7021-ccb785d8c-ztcqg"
Sep 18 14:11:35.670805 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:35.670774    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bee7612-56a4-4f90-a62a-8a7674f5634d-maas-api-tls\") pod \"maas-api-e2e-autores-ae7021-ccb785d8c-ztcqg\" (UID: \"0bee7612-56a4-4f90-a62a-8a7674f5634d\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ae7021-ccb785d8c-ztcqg"
Sep 18 14:11:35.670952 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:11:35.670856    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-ae7021-serving-cert: secret "maas-api-e2e-autores-ae7021-serving-cert" not found
Sep 18 14:11:35.670952 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:11:35.670909    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bee7612-56a4-4f90-a62a-8a7674f5634d-maas-api-tls podName:0bee7612-56a4-4f90-a62a-8a7674f5634d nodeName:}" failed. No retries permitted until 2026-09-18 14:11:36.170891552 +0000 UTC m=+768.059607464 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0bee7612-56a4-4f90-a62a-8a7674f5634d-maas-api-tls") pod "maas-api-e2e-autores-ae7021-ccb785d8c-ztcqg" (UID: "0bee7612-56a4-4f90-a62a-8a7674f5634d") : secret "maas-api-e2e-autores-ae7021-serving-cert" not found
Sep 18 14:11:35.671378 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:35.671359    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bee7612-56a4-4f90-a62a-8a7674f5634d-openshift-service-ca\") pod \"maas-api-e2e-autores-ae7021-ccb785d8c-ztcqg\" (UID: \"0bee7612-56a4-4f90-a62a-8a7674f5634d\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ae7021-ccb785d8c-ztcqg"
Sep 18 14:11:35.673094 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:35.673075    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bee7612-56a4-4f90-a62a-8a7674f5634d-metrics-certs\") pod \"maas-api-e2e-autores-ae7021-ccb785d8c-ztcqg\" (UID: \"0bee7612-56a4-4f90-a62a-8a7674f5634d\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ae7021-ccb785d8c-ztcqg"
Sep 18 14:11:35.679258 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:35.679239    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z6ns\" (UniqueName: \"kubernetes.io/projected/0bee7612-56a4-4f90-a62a-8a7674f5634d-kube-api-access-4z6ns\") pod \"maas-api-e2e-autores-ae7021-ccb785d8c-ztcqg\" (UID: \"0bee7612-56a4-4f90-a62a-8a7674f5634d\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ae7021-ccb785d8c-ztcqg"
Sep 18 14:11:35.774414 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:35.774352    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:36.113643 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:36.113613    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:36.174663 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:36.174638    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bee7612-56a4-4f90-a62a-8a7674f5634d-maas-api-tls\") pod \"maas-api-e2e-autores-ae7021-ccb785d8c-ztcqg\" (UID: \"0bee7612-56a4-4f90-a62a-8a7674f5634d\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ae7021-ccb785d8c-ztcqg"
Sep 18 14:11:36.177049 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:36.177021    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bee7612-56a4-4f90-a62a-8a7674f5634d-maas-api-tls\") pod \"maas-api-e2e-autores-ae7021-ccb785d8c-ztcqg\" (UID: \"0bee7612-56a4-4f90-a62a-8a7674f5634d\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ae7021-ccb785d8c-ztcqg"
Sep 18 14:11:36.214470 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:36.214442    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:36.417549 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:36.417460    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ae7021-ccb785d8c-ztcqg"
Sep 18 14:11:36.654357 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:36.654307    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:36.709379 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:36.708162    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-ae7021-ccb785d8c-ztcqg"]
Sep 18 14:11:36.713084 ip-10-0-143-120 kubenswrapper[2579]: W0918 14:11:36.713034    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bee7612_56a4_4f90_a62a_8a7674f5634d.slice/crio-b0879fbf10f30f039eef6439125c5400211e453d616f8cb87f32b15a88e1c010 WatchSource:0}: Error finding container b0879fbf10f30f039eef6439125c5400211e453d616f8cb87f32b15a88e1c010: Status 404 returned error can't find the container with id b0879fbf10f30f039eef6439125c5400211e453d616f8cb87f32b15a88e1c010
Sep 18 14:11:37.132351 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:37.132311    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ae7021-ccb785d8c-ztcqg" event={"ID":"0bee7612-56a4-4f90-a62a-8a7674f5634d","Type":"ContainerStarted","Data":"b0879fbf10f30f039eef6439125c5400211e453d616f8cb87f32b15a88e1c010"}
Sep 18 14:11:37.231220 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:37.231192    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:37.387444 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:37.387358    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:37.499032 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:37.499001    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:37.634388 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:37.634355    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:38.027591 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:38.027553    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:38.140093 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:38.140049    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ae7021-ccb785d8c-ztcqg" event={"ID":"0bee7612-56a4-4f90-a62a-8a7674f5634d","Type":"ContainerStarted","Data":"78f22e5c9aa3a88dd975629be474aaf3e4f026833fa0aa7b8d3c31da044b2efa"}
Sep 18 14:11:38.140280 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:38.140145    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ae7021-ccb785d8c-ztcqg"
Sep 18 14:11:38.160755 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:38.160695    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ae7021-ccb785d8c-ztcqg" podStartSLOduration=2.742142898 podStartE2EDuration="3.160678369s" totalImagesPullingTime="418.535471ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:11:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:11:38.158452839 +0000 UTC m=+770.047168771" watchObservedRunningTime="2026-09-18 14:11:38.160678369 +0000 UTC m=+770.049394338"
Sep 18 14:11:38.817366 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:38.816862    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:39.413085 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:39.413049    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:39.516028 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:39.515999    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:39.728201 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:39.728112    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:39.913555 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:39.913526    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:40.038691 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:40.038618    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:40.171891 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:40.171857    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:40.351224 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:40.351190    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:41.013628 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:41.013588    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:41.846268 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:41.846239    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:41.943868 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:41.943839    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:42.159898 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:42.159821    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:42.473117 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:42.473035    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:42.748351 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:42.748262    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:43.031641 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:43.031562    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:43.137918 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:43.137886    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:43.923889 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:43.923843    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:44.027790 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:44.027756    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:44.150426 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:44.150395    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ae7021-ccb785d8c-ztcqg"
Sep 18 14:11:44.221366 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:44.221288    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:45.217430 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:45.217396    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:45.634006 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:45.633972    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:45.747132 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:45.747092    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:46.516361 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:46.516327    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:48.092660 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:48.092632    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:48.198589 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:48.198556    2579 generic.go:356] "Generic (PLEG): container finished" podID="1cefcc6a-27d6-4530-821e-a3c1f0272cd8" containerID="9e51caa5b3fb538fa2377e893fae057d1d09cc648b31165692b33dce1f3133c4" exitCode=0
Sep 18 14:11:48.198766 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:48.198642    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m" event={"ID":"1cefcc6a-27d6-4530-821e-a3c1f0272cd8","Type":"ContainerDied","Data":"9e51caa5b3fb538fa2377e893fae057d1d09cc648b31165692b33dce1f3133c4"}
Sep 18 14:11:48.199205 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:48.199164    2579 scope.go:117] "RemoveContainer" containerID="9e51caa5b3fb538fa2377e893fae057d1d09cc648b31165692b33dce1f3133c4"
Sep 18 14:11:48.200440 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:48.200415    2579 generic.go:356] "Generic (PLEG): container finished" podID="aebd26b0-fc86-42ed-b073-15eddb601b91" containerID="a4828280ba2425af30941e863a455d11397546946b1dfd655ff686b2fa931ba4" exitCode=0
Sep 18 14:11:48.200531 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:48.200490    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh" event={"ID":"aebd26b0-fc86-42ed-b073-15eddb601b91","Type":"ContainerDied","Data":"a4828280ba2425af30941e863a455d11397546946b1dfd655ff686b2fa931ba4"}
Sep 18 14:11:48.200531 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:48.200494    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:11:48.200610 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:48.200559    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:11:48.200610 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:48.200590    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:11:48.200832 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:48.200819    2579 scope.go:117] "RemoveContainer" containerID="a4828280ba2425af30941e863a455d11397546946b1dfd655ff686b2fa931ba4"
Sep 18 14:11:48.201987 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:48.201956    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:11:48.202067 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:48.202037    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:11:48.202135 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:48.202088    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:11:49.212319 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:49.211597    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh" event={"ID":"aebd26b0-fc86-42ed-b073-15eddb601b91","Type":"ContainerStarted","Data":"6456a1684d9b6480662f56507987a6d2772291995bdd3aa2b92b32275d950b26"}
Sep 18 14:11:49.214681 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:49.214651    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m" event={"ID":"1cefcc6a-27d6-4530-821e-a3c1f0272cd8","Type":"ContainerStarted","Data":"64177fbd8210d20c9dc335cac1a8afc53be0d494342037d21d04146d430380b1"}
Sep 18 14:11:49.390845 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:49.390421    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m"
Sep 18 14:11:49.390845 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:49.390543    2579 patch_prober.go:28] interesting pod/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 18 14:11:49.390845 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:49.390586    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m" podUID="1cefcc6a-27d6-4530-821e-a3c1f0272cd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 18 14:11:49.770029 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:49.769981    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ee8a08-9bf55585f-sbtvp"]
Sep 18 14:11:49.782300 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:49.782251    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee8a08-9bf55585f-sbtvp"
Sep 18 14:11:49.789104 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:49.788568    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-ee8a08-serving-cert\""
Sep 18 14:11:49.812919 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:49.812891    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ee8a08-9bf55585f-sbtvp"]
Sep 18 14:11:49.819425 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:49.819399    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh"
Sep 18 14:11:49.820386 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:49.820355    2579 patch_prober.go:28] interesting pod/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Sep 18 14:11:49.820473 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:49.820417    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh" podUID="aebd26b0-fc86-42ed-b073-15eddb601b91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Sep 18 14:11:49.871839 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:49.871789    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:49.912198 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:49.910643    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3358232d-d8be-48ee-8ee4-3e146330e3ec-maas-api-tls\") pod \"maas-api-e2e-inf-b-ee8a08-9bf55585f-sbtvp\" (UID: \"3358232d-d8be-48ee-8ee4-3e146330e3ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee8a08-9bf55585f-sbtvp"
Sep 18 14:11:49.912198 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:49.910691    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3358232d-d8be-48ee-8ee4-3e146330e3ec-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ee8a08-9bf55585f-sbtvp\" (UID: \"3358232d-d8be-48ee-8ee4-3e146330e3ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee8a08-9bf55585f-sbtvp"
Sep 18 14:11:49.912198 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:49.910776    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3358232d-d8be-48ee-8ee4-3e146330e3ec-metrics-certs\") pod \"maas-api-e2e-inf-b-ee8a08-9bf55585f-sbtvp\" (UID: \"3358232d-d8be-48ee-8ee4-3e146330e3ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee8a08-9bf55585f-sbtvp"
Sep 18 14:11:49.912198 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:49.910826    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbfts\" (UniqueName: \"kubernetes.io/projected/3358232d-d8be-48ee-8ee4-3e146330e3ec-kube-api-access-kbfts\") pod \"maas-api-e2e-inf-b-ee8a08-9bf55585f-sbtvp\" (UID: \"3358232d-d8be-48ee-8ee4-3e146330e3ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee8a08-9bf55585f-sbtvp"
Sep 18 14:11:50.011843 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:50.011809    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbfts\" (UniqueName: \"kubernetes.io/projected/3358232d-d8be-48ee-8ee4-3e146330e3ec-kube-api-access-kbfts\") pod \"maas-api-e2e-inf-b-ee8a08-9bf55585f-sbtvp\" (UID: \"3358232d-d8be-48ee-8ee4-3e146330e3ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee8a08-9bf55585f-sbtvp"
Sep 18 14:11:50.012197 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:50.012156    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3358232d-d8be-48ee-8ee4-3e146330e3ec-maas-api-tls\") pod \"maas-api-e2e-inf-b-ee8a08-9bf55585f-sbtvp\" (UID: \"3358232d-d8be-48ee-8ee4-3e146330e3ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee8a08-9bf55585f-sbtvp"
Sep 18 14:11:50.012343 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:50.012327    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3358232d-d8be-48ee-8ee4-3e146330e3ec-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ee8a08-9bf55585f-sbtvp\" (UID: \"3358232d-d8be-48ee-8ee4-3e146330e3ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee8a08-9bf55585f-sbtvp"
Sep 18 14:11:50.012525 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:50.012509    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3358232d-d8be-48ee-8ee4-3e146330e3ec-metrics-certs\") pod \"maas-api-e2e-inf-b-ee8a08-9bf55585f-sbtvp\" (UID: \"3358232d-d8be-48ee-8ee4-3e146330e3ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee8a08-9bf55585f-sbtvp"
Sep 18 14:11:50.013046 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:50.013024    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3358232d-d8be-48ee-8ee4-3e146330e3ec-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ee8a08-9bf55585f-sbtvp\" (UID: \"3358232d-d8be-48ee-8ee4-3e146330e3ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee8a08-9bf55585f-sbtvp"
Sep 18 14:11:50.015568 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:50.015549    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3358232d-d8be-48ee-8ee4-3e146330e3ec-metrics-certs\") pod \"maas-api-e2e-inf-b-ee8a08-9bf55585f-sbtvp\" (UID: \"3358232d-d8be-48ee-8ee4-3e146330e3ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee8a08-9bf55585f-sbtvp"
Sep 18 14:11:50.039787 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:50.039713    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-ee8a08-serving-cert\""
Sep 18 14:11:50.041817 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:50.041792    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbfts\" (UniqueName: \"kubernetes.io/projected/3358232d-d8be-48ee-8ee4-3e146330e3ec-kube-api-access-kbfts\") pod \"maas-api-e2e-inf-b-ee8a08-9bf55585f-sbtvp\" (UID: \"3358232d-d8be-48ee-8ee4-3e146330e3ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee8a08-9bf55585f-sbtvp"
Sep 18 14:11:50.044380 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:50.044356    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3358232d-d8be-48ee-8ee4-3e146330e3ec-maas-api-tls\") pod \"maas-api-e2e-inf-b-ee8a08-9bf55585f-sbtvp\" (UID: \"3358232d-d8be-48ee-8ee4-3e146330e3ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee8a08-9bf55585f-sbtvp"
Sep 18 14:11:50.113699 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:50.113660    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee8a08-9bf55585f-sbtvp"
Sep 18 14:11:50.148309 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:50.148270    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:50.393071 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:50.392608    2579 patch_prober.go:28] interesting pod/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 18 14:11:50.393071 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:50.392674    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m" podUID="1cefcc6a-27d6-4530-821e-a3c1f0272cd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 18 14:11:50.438232 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:50.438204    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:50.493333 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:50.493303    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ee8a08-9bf55585f-sbtvp"]
Sep 18 14:11:50.818969 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:50.818913    2579 patch_prober.go:28] interesting pod/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Sep 18 14:11:50.819131 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:50.818975    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh" podUID="aebd26b0-fc86-42ed-b073-15eddb601b91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Sep 18 14:11:51.055418 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:51.055380    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh"
Sep 18 14:11:51.227352 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:51.227270    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee8a08-9bf55585f-sbtvp" event={"ID":"3358232d-d8be-48ee-8ee4-3e146330e3ec","Type":"ContainerStarted","Data":"f031a54f16aa89737fecd1cff95f954fd516fc97316a5dbec0dac9e38ffe26c3"}
Sep 18 14:11:51.227352 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:51.227327    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee8a08-9bf55585f-sbtvp" event={"ID":"3358232d-d8be-48ee-8ee4-3e146330e3ec","Type":"ContainerStarted","Data":"83f0e9b654103591368b23513c59236ae7b721a061281fe06e9ff4eb22f90d9e"}
Sep 18 14:11:51.227663 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:51.227647    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee8a08-9bf55585f-sbtvp"
Sep 18 14:11:51.322776 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:51.322720    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee8a08-9bf55585f-sbtvp" podStartSLOduration=1.895383925 podStartE2EDuration="2.322703282s" totalImagesPullingTime="427.319357ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:11:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:11:51.320609993 +0000 UTC m=+783.209325923" watchObservedRunningTime="2026-09-18 14:11:51.322703282 +0000 UTC m=+783.211419212"
Sep 18 14:11:51.391254 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:51.391219    2579 patch_prober.go:28] interesting pod/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 18 14:11:51.391537 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:51.391499    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m" podUID="1cefcc6a-27d6-4530-821e-a3c1f0272cd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 18 14:11:51.443242 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:51.443217    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:51.747386 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:51.747359    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:51.819701 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:51.819669    2579 patch_prober.go:28] interesting pod/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Sep 18 14:11:51.819866 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:51.819733    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh" podUID="aebd26b0-fc86-42ed-b073-15eddb601b91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Sep 18 14:11:52.375874 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:52.375800    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:52.390530 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:52.390501    2579 patch_prober.go:28] interesting pod/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 18 14:11:52.390758 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:52.390732    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m" podUID="1cefcc6a-27d6-4530-821e-a3c1f0272cd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 18 14:11:52.819673 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:52.819639    2579 patch_prober.go:28] interesting pod/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Sep 18 14:11:52.820047 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:52.819690    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh" podUID="aebd26b0-fc86-42ed-b073-15eddb601b91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Sep 18 14:11:53.390586 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:53.390547    2579 patch_prober.go:28] interesting pod/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 18 14:11:53.390866 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:53.390834    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m" podUID="1cefcc6a-27d6-4530-821e-a3c1f0272cd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 18 14:11:53.613734 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:53.613684    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:53.819493 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:53.819454    2579 patch_prober.go:28] interesting pod/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Sep 18 14:11:53.819675 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:53.819513    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh" podUID="aebd26b0-fc86-42ed-b073-15eddb601b91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Sep 18 14:11:54.470476 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:54.470433    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m"
Sep 18 14:11:54.470948 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:54.470859    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m"
Sep 18 14:11:54.472017 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:54.471993    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m"
Sep 18 14:11:54.653917 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:54.653884    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:54.790819 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:54.790742    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:54.822712 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:54.822691    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh"
Sep 18 14:11:54.823820 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:54.823805    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh"
Sep 18 14:11:54.903249 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:54.903221    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:55.023523 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:55.023490    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:55.188858 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:55.188828    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:55.482151 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:55.482074    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:55.826418 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:55.826390    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:56.782500 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:56.782466    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:57.237590 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:57.237565    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee8a08-9bf55585f-sbtvp"
Sep 18 14:11:57.604496 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:57.604461    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:58.530532 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:58.530503    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:11:59.529784 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:11:59.529750    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:00.393976 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:00.393937    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:01.287342 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:01.287291    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:02.473402 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:02.473367    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:03.453784 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:03.453748    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:03.843523 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:03.843494    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:04.370246 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:04.370210    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:04.479624 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:04.479595    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:05.335151 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:05.335117    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:06.000944 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:06.000915    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:06.099953 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:06.099927    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:06.179478 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:06.179437    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:06.277661 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:06.277587    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:06.427543 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:06.427513    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:06.565841 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:06.565811    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m"]
Sep 18 14:12:06.566031 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:06.565995    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m" podUID="1cefcc6a-27d6-4530-821e-a3c1f0272cd8" containerName="istio-proxy" containerID="cri-o://64177fbd8210d20c9dc335cac1a8afc53be0d494342037d21d04146d430380b1" gracePeriod=30
Sep 18 14:12:06.798839 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:06.798801    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:07.228623 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:07.228589    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:07.384345 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:07.384317    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:07.880390 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:07.880363    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:08.277897 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:08.277776    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:08.446312 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:08.446278    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:09.472925 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:09.472881    2579 patch_prober.go:28] interesting pod/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 14:12:09.473392 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:09.472931    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m" podUID="1cefcc6a-27d6-4530-821e-a3c1f0272cd8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 14:12:10.019085 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:10.019042    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:10.758531 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:10.758500    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:11.023218 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:11.023143    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:11.448043 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:11.448007    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:11.820728 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:11.820703    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m"
Sep 18 14:12:11.906946 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:11.906917    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lwvhl\" (UniqueName: \"kubernetes.io/projected/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-kube-api-access-lwvhl\") pod \"1cefcc6a-27d6-4530-821e-a3c1f0272cd8\" (UID: \"1cefcc6a-27d6-4530-821e-a3c1f0272cd8\") "
Sep 18 14:12:11.907136 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:11.906967    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-istio-data\") pod \"1cefcc6a-27d6-4530-821e-a3c1f0272cd8\" (UID: \"1cefcc6a-27d6-4530-821e-a3c1f0272cd8\") "
Sep 18 14:12:11.907136 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:11.907002    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-workload-socket\") pod \"1cefcc6a-27d6-4530-821e-a3c1f0272cd8\" (UID: \"1cefcc6a-27d6-4530-821e-a3c1f0272cd8\") "
Sep 18 14:12:11.907136 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:11.907063    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-workload-certs\") pod \"1cefcc6a-27d6-4530-821e-a3c1f0272cd8\" (UID: \"1cefcc6a-27d6-4530-821e-a3c1f0272cd8\") "
Sep 18 14:12:11.907136 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:11.907101    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-istiod-ca-cert\") pod \"1cefcc6a-27d6-4530-821e-a3c1f0272cd8\" (UID: \"1cefcc6a-27d6-4530-821e-a3c1f0272cd8\") "
Sep 18 14:12:11.907385 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:11.907137    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-credential-socket\") pod \"1cefcc6a-27d6-4530-821e-a3c1f0272cd8\" (UID: \"1cefcc6a-27d6-4530-821e-a3c1f0272cd8\") "
Sep 18 14:12:11.907385 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:11.907199    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-istio-token\") pod \"1cefcc6a-27d6-4530-821e-a3c1f0272cd8\" (UID: \"1cefcc6a-27d6-4530-821e-a3c1f0272cd8\") "
Sep 18 14:12:11.907385 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:11.907246    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-istio-podinfo\") pod \"1cefcc6a-27d6-4530-821e-a3c1f0272cd8\" (UID: \"1cefcc6a-27d6-4530-821e-a3c1f0272cd8\") "
Sep 18 14:12:11.907385 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:11.907283    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-istio-envoy\") pod \"1cefcc6a-27d6-4530-821e-a3c1f0272cd8\" (UID: \"1cefcc6a-27d6-4530-821e-a3c1f0272cd8\") "
Sep 18 14:12:11.907385 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:11.907340    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-istio-data" (OuterVolumeSpecName: "istio-data") pod "1cefcc6a-27d6-4530-821e-a3c1f0272cd8" (UID: "1cefcc6a-27d6-4530-821e-a3c1f0272cd8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:12:11.907648 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:11.907364    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1cefcc6a-27d6-4530-821e-a3c1f0272cd8" (UID: "1cefcc6a-27d6-4530-821e-a3c1f0272cd8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:12:11.907648 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:11.907506    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1cefcc6a-27d6-4530-821e-a3c1f0272cd8" (UID: "1cefcc6a-27d6-4530-821e-a3c1f0272cd8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:12:11.907648 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:11.907614    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:12:11.907648 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:11.907634    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:12:11.907648 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:11.907650    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:12:11.907906 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:11.907804    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1cefcc6a-27d6-4530-821e-a3c1f0272cd8" (UID: "1cefcc6a-27d6-4530-821e-a3c1f0272cd8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:12:11.907906 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:11.907852    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1cefcc6a-27d6-4530-821e-a3c1f0272cd8" (UID: "1cefcc6a-27d6-4530-821e-a3c1f0272cd8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:12:11.909415 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:11.909389    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-istio-token" (OuterVolumeSpecName: "istio-token") pod "1cefcc6a-27d6-4530-821e-a3c1f0272cd8" (UID: "1cefcc6a-27d6-4530-821e-a3c1f0272cd8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:12:11.909520 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:11.909427    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1cefcc6a-27d6-4530-821e-a3c1f0272cd8" (UID: "1cefcc6a-27d6-4530-821e-a3c1f0272cd8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:12:11.909627 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:11.909612    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-kube-api-access-lwvhl" (OuterVolumeSpecName: "kube-api-access-lwvhl") pod "1cefcc6a-27d6-4530-821e-a3c1f0272cd8" (UID: "1cefcc6a-27d6-4530-821e-a3c1f0272cd8"). InnerVolumeSpecName "kube-api-access-lwvhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:12:11.909726 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:11.909711    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1cefcc6a-27d6-4530-821e-a3c1f0272cd8" (UID: "1cefcc6a-27d6-4530-821e-a3c1f0272cd8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 14:12:12.008505 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:12.008454    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:12:12.008505 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:12.008473    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwvhl\" (UniqueName: \"kubernetes.io/projected/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-kube-api-access-lwvhl\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:12:12.008505 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:12.008483    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:12:12.008505 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:12.008493    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:12:12.008505 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:12.008500    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:12:12.008505 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:12.008508    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cefcc6a-27d6-4530-821e-a3c1f0272cd8-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:12:12.326274 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:12.326245    2579 generic.go:356] "Generic (PLEG): container finished" podID="1cefcc6a-27d6-4530-821e-a3c1f0272cd8" containerID="64177fbd8210d20c9dc335cac1a8afc53be0d494342037d21d04146d430380b1" exitCode=0
Sep 18 14:12:12.326426 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:12.326289    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m" event={"ID":"1cefcc6a-27d6-4530-821e-a3c1f0272cd8","Type":"ContainerDied","Data":"64177fbd8210d20c9dc335cac1a8afc53be0d494342037d21d04146d430380b1"}
Sep 18 14:12:12.326426 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:12.326313    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m"
Sep 18 14:12:12.326426 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:12.326329    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m" event={"ID":"1cefcc6a-27d6-4530-821e-a3c1f0272cd8","Type":"ContainerDied","Data":"79cd28e65b8a7e12a1257cad0888ea543b307b26c78d5c322fc82bec21e1b374"}
Sep 18 14:12:12.326426 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:12.326350    2579 scope.go:117] "RemoveContainer" containerID="64177fbd8210d20c9dc335cac1a8afc53be0d494342037d21d04146d430380b1"
Sep 18 14:12:12.337142 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:12.337116    2579 scope.go:117] "RemoveContainer" containerID="9e51caa5b3fb538fa2377e893fae057d1d09cc648b31165692b33dce1f3133c4"
Sep 18 14:12:12.344967 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:12.344946    2579 scope.go:117] "RemoveContainer" containerID="64177fbd8210d20c9dc335cac1a8afc53be0d494342037d21d04146d430380b1"
Sep 18 14:12:12.345248 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:12:12.345224    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64177fbd8210d20c9dc335cac1a8afc53be0d494342037d21d04146d430380b1\": container with ID starting with 64177fbd8210d20c9dc335cac1a8afc53be0d494342037d21d04146d430380b1 not found: ID does not exist" containerID="64177fbd8210d20c9dc335cac1a8afc53be0d494342037d21d04146d430380b1"
Sep 18 14:12:12.345299 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:12.345257    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64177fbd8210d20c9dc335cac1a8afc53be0d494342037d21d04146d430380b1"} err="failed to get container status \"64177fbd8210d20c9dc335cac1a8afc53be0d494342037d21d04146d430380b1\": rpc error: code = NotFound desc = could not find container \"64177fbd8210d20c9dc335cac1a8afc53be0d494342037d21d04146d430380b1\": container with ID starting with 64177fbd8210d20c9dc335cac1a8afc53be0d494342037d21d04146d430380b1 not found: ID does not exist"
Sep 18 14:12:12.345299 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:12.345274    2579 scope.go:117] "RemoveContainer" containerID="9e51caa5b3fb538fa2377e893fae057d1d09cc648b31165692b33dce1f3133c4"
Sep 18 14:12:12.345535 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:12:12.345516    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e51caa5b3fb538fa2377e893fae057d1d09cc648b31165692b33dce1f3133c4\": container with ID starting with 9e51caa5b3fb538fa2377e893fae057d1d09cc648b31165692b33dce1f3133c4 not found: ID does not exist" containerID="9e51caa5b3fb538fa2377e893fae057d1d09cc648b31165692b33dce1f3133c4"
Sep 18 14:12:12.345577 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:12.345541    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e51caa5b3fb538fa2377e893fae057d1d09cc648b31165692b33dce1f3133c4"} err="failed to get container status \"9e51caa5b3fb538fa2377e893fae057d1d09cc648b31165692b33dce1f3133c4\": rpc error: code = NotFound desc = could not find container \"9e51caa5b3fb538fa2377e893fae057d1d09cc648b31165692b33dce1f3133c4\": container with ID starting with 9e51caa5b3fb538fa2377e893fae057d1d09cc648b31165692b33dce1f3133c4 not found: ID does not exist"
Sep 18 14:12:12.352157 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:12.352134    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m"]
Sep 18 14:12:12.360806 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:12.360778    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-eeb0a3f4-openshift-default-77b7b5576-db42m"]
Sep 18 14:12:12.482361 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:12.482337    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:13.321838 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:13.321806    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:14.036646 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:14.036610    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cefcc6a-27d6-4530-821e-a3c1f0272cd8" path="/var/lib/kubelet/pods/1cefcc6a-27d6-4530-821e-a3c1f0272cd8/volumes"
Sep 18 14:12:14.038548 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:14.038522    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:14.713481 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:14.713451    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5f7e430d-885794f4f-nj9dn"]
Sep 18 14:12:14.714072 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:14.714049    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cefcc6a-27d6-4530-821e-a3c1f0272cd8" containerName="istio-proxy"
Sep 18 14:12:14.714200 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:14.714075    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cefcc6a-27d6-4530-821e-a3c1f0272cd8" containerName="istio-proxy"
Sep 18 14:12:14.714265 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:14.714218    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cefcc6a-27d6-4530-821e-a3c1f0272cd8" containerName="istio-proxy"
Sep 18 14:12:14.714265 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:14.714234    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cefcc6a-27d6-4530-821e-a3c1f0272cd8" containerName="istio-proxy"
Sep 18 14:12:14.718991 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:14.718965    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f7e430d-885794f4f-nj9dn"
Sep 18 14:12:14.721307 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:14.721280    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-5f7e430d-serving-cert\""
Sep 18 14:12:14.730232 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:14.730211    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5f7e430d-885794f4f-nj9dn"]
Sep 18 14:12:14.750976 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:14.750942    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:14.814875 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:14.814835    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:14.832137 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:14.832108    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b4b4f78-fa7b-4d03-8fee-dfea18872a95-metrics-certs\") pod \"maas-api-e2e-migrate-5f7e430d-885794f4f-nj9dn\" (UID: \"5b4b4f78-fa7b-4d03-8fee-dfea18872a95\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f7e430d-885794f4f-nj9dn"
Sep 18 14:12:14.832265 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:14.832163    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b4b4f78-fa7b-4d03-8fee-dfea18872a95-maas-api-tls\") pod \"maas-api-e2e-migrate-5f7e430d-885794f4f-nj9dn\" (UID: \"5b4b4f78-fa7b-4d03-8fee-dfea18872a95\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f7e430d-885794f4f-nj9dn"
Sep 18 14:12:14.832265 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:14.832211    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b4b4f78-fa7b-4d03-8fee-dfea18872a95-openshift-service-ca\") pod \"maas-api-e2e-migrate-5f7e430d-885794f4f-nj9dn\" (UID: \"5b4b4f78-fa7b-4d03-8fee-dfea18872a95\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f7e430d-885794f4f-nj9dn"
Sep 18 14:12:14.832265 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:14.832234    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhcv2\" (UniqueName: \"kubernetes.io/projected/5b4b4f78-fa7b-4d03-8fee-dfea18872a95-kube-api-access-mhcv2\") pod \"maas-api-e2e-migrate-5f7e430d-885794f4f-nj9dn\" (UID: \"5b4b4f78-fa7b-4d03-8fee-dfea18872a95\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f7e430d-885794f4f-nj9dn"
Sep 18 14:12:14.932734 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:14.932701    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b4b4f78-fa7b-4d03-8fee-dfea18872a95-metrics-certs\") pod \"maas-api-e2e-migrate-5f7e430d-885794f4f-nj9dn\" (UID: \"5b4b4f78-fa7b-4d03-8fee-dfea18872a95\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f7e430d-885794f4f-nj9dn"
Sep 18 14:12:14.932904 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:14.932758    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b4b4f78-fa7b-4d03-8fee-dfea18872a95-maas-api-tls\") pod \"maas-api-e2e-migrate-5f7e430d-885794f4f-nj9dn\" (UID: \"5b4b4f78-fa7b-4d03-8fee-dfea18872a95\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f7e430d-885794f4f-nj9dn"
Sep 18 14:12:14.932904 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:14.932780    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b4b4f78-fa7b-4d03-8fee-dfea18872a95-openshift-service-ca\") pod \"maas-api-e2e-migrate-5f7e430d-885794f4f-nj9dn\" (UID: \"5b4b4f78-fa7b-4d03-8fee-dfea18872a95\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f7e430d-885794f4f-nj9dn"
Sep 18 14:12:14.932904 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:14.932798    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhcv2\" (UniqueName: \"kubernetes.io/projected/5b4b4f78-fa7b-4d03-8fee-dfea18872a95-kube-api-access-mhcv2\") pod \"maas-api-e2e-migrate-5f7e430d-885794f4f-nj9dn\" (UID: \"5b4b4f78-fa7b-4d03-8fee-dfea18872a95\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f7e430d-885794f4f-nj9dn"
Sep 18 14:12:14.933010 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:12:14.932923    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-5f7e430d-serving-cert: secret "maas-api-e2e-migrate-5f7e430d-serving-cert" not found
Sep 18 14:12:14.933010 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:12:14.933001    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b4b4f78-fa7b-4d03-8fee-dfea18872a95-maas-api-tls podName:5b4b4f78-fa7b-4d03-8fee-dfea18872a95 nodeName:}" failed. No retries permitted until 2026-09-18 14:12:15.43298468 +0000 UTC m=+807.321700589 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5b4b4f78-fa7b-4d03-8fee-dfea18872a95-maas-api-tls") pod "maas-api-e2e-migrate-5f7e430d-885794f4f-nj9dn" (UID: "5b4b4f78-fa7b-4d03-8fee-dfea18872a95") : secret "maas-api-e2e-migrate-5f7e430d-serving-cert" not found
Sep 18 14:12:14.933572 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:14.933555    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b4b4f78-fa7b-4d03-8fee-dfea18872a95-openshift-service-ca\") pod \"maas-api-e2e-migrate-5f7e430d-885794f4f-nj9dn\" (UID: \"5b4b4f78-fa7b-4d03-8fee-dfea18872a95\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f7e430d-885794f4f-nj9dn"
Sep 18 14:12:14.935279 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:14.935262    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b4b4f78-fa7b-4d03-8fee-dfea18872a95-metrics-certs\") pod \"maas-api-e2e-migrate-5f7e430d-885794f4f-nj9dn\" (UID: \"5b4b4f78-fa7b-4d03-8fee-dfea18872a95\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f7e430d-885794f4f-nj9dn"
Sep 18 14:12:14.942677 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:14.942655    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhcv2\" (UniqueName: \"kubernetes.io/projected/5b4b4f78-fa7b-4d03-8fee-dfea18872a95-kube-api-access-mhcv2\") pod \"maas-api-e2e-migrate-5f7e430d-885794f4f-nj9dn\" (UID: \"5b4b4f78-fa7b-4d03-8fee-dfea18872a95\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f7e430d-885794f4f-nj9dn"
Sep 18 14:12:15.067931 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:15.067896    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:15.390875 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:15.390801    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:15.436817 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:15.436790    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b4b4f78-fa7b-4d03-8fee-dfea18872a95-maas-api-tls\") pod \"maas-api-e2e-migrate-5f7e430d-885794f4f-nj9dn\" (UID: \"5b4b4f78-fa7b-4d03-8fee-dfea18872a95\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f7e430d-885794f4f-nj9dn"
Sep 18 14:12:15.439065 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:15.439043    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b4b4f78-fa7b-4d03-8fee-dfea18872a95-maas-api-tls\") pod \"maas-api-e2e-migrate-5f7e430d-885794f4f-nj9dn\" (UID: \"5b4b4f78-fa7b-4d03-8fee-dfea18872a95\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f7e430d-885794f4f-nj9dn"
Sep 18 14:12:15.491712 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:15.491685    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:15.631378 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:15.631353    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f7e430d-885794f4f-nj9dn"
Sep 18 14:12:15.913882 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:15.913847    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5f7e430d-885794f4f-nj9dn"]
Sep 18 14:12:15.916186 ip-10-0-143-120 kubenswrapper[2579]: W0918 14:12:15.916146    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b4b4f78_fa7b_4d03_8fee_dfea18872a95.slice/crio-ccc1559f09a03d71f5f52789c67cfe4c6fa36c80a478d3abf0b424fe5c600ae6 WatchSource:0}: Error finding container ccc1559f09a03d71f5f52789c67cfe4c6fa36c80a478d3abf0b424fe5c600ae6: Status 404 returned error can't find the container with id ccc1559f09a03d71f5f52789c67cfe4c6fa36c80a478d3abf0b424fe5c600ae6
Sep 18 14:12:16.126298 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:16.126270    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:16.345941 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:16.345907    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f7e430d-885794f4f-nj9dn" event={"ID":"5b4b4f78-fa7b-4d03-8fee-dfea18872a95","Type":"ContainerStarted","Data":"ccc1559f09a03d71f5f52789c67cfe4c6fa36c80a478d3abf0b424fe5c600ae6"}
Sep 18 14:12:16.861687 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:16.861656    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:17.351801 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:17.351768    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f7e430d-885794f4f-nj9dn" event={"ID":"5b4b4f78-fa7b-4d03-8fee-dfea18872a95","Type":"ContainerStarted","Data":"deb4e162e7193326add9cfb165e88673a6e54efccfcd0a045035578e01b0e996"}
Sep 18 14:12:17.352268 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:17.351861    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f7e430d-885794f4f-nj9dn"
Sep 18 14:12:17.375561 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:17.375514    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f7e430d-885794f4f-nj9dn" podStartSLOduration=2.930438863 podStartE2EDuration="3.375497871s" totalImagesPullingTime="445.059008ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:12:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:12:17.373937364 +0000 UTC m=+809.262653296" watchObservedRunningTime="2026-09-18 14:12:17.375497871 +0000 UTC m=+809.264213803"
Sep 18 14:12:17.472281 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:17.472251    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:17.929011 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:17.928973    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:19.485341 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:19.485303    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:23.363072 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:23.363043    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f7e430d-885794f4f-nj9dn"
Sep 18 14:12:33.945339 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:33.945301    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-ae7021/autores-model-ae7021-kserve-7c4c8c4ff5-l77hk"]
Sep 18 14:12:33.945937 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:33.945803    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cefcc6a-27d6-4530-821e-a3c1f0272cd8" containerName="istio-proxy"
Sep 18 14:12:33.945937 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:33.945823    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cefcc6a-27d6-4530-821e-a3c1f0272cd8" containerName="istio-proxy"
Sep 18 14:12:33.950273 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:33.950251    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-ae7021/autores-model-ae7021-kserve-7c4c8c4ff5-l77hk"
Sep 18 14:12:33.977719 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:33.977695    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-ae7021\"/\"openshift-service-ca.crt\""
Sep 18 14:12:33.977826 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:33.977787    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-ae7021\"/\"autores-model-ae7021-kserve-self-signed-certs\""
Sep 18 14:12:33.978043 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:33.977837    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-ae7021\"/\"default-dockercfg-cfmnj\""
Sep 18 14:12:33.978043 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:33.977846    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-ae7021\"/\"kube-root-ca.crt\""
Sep 18 14:12:33.979963 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:33.979945    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-ae7021/autores-model-ae7021-kserve-7c4c8c4ff5-l77hk"]
Sep 18 14:12:33.992039 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:33.992017    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b2fa854-3ea0-482b-b421-dc06a95803dc-dshm\") pod \"autores-model-ae7021-kserve-7c4c8c4ff5-l77hk\" (UID: \"1b2fa854-3ea0-482b-b421-dc06a95803dc\") " pod="ai-tenant-e2e-autores-ae7021/autores-model-ae7021-kserve-7c4c8c4ff5-l77hk"
Sep 18 14:12:33.992166 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:33.992059    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b2fa854-3ea0-482b-b421-dc06a95803dc-tls-certs\") pod \"autores-model-ae7021-kserve-7c4c8c4ff5-l77hk\" (UID: \"1b2fa854-3ea0-482b-b421-dc06a95803dc\") " pod="ai-tenant-e2e-autores-ae7021/autores-model-ae7021-kserve-7c4c8c4ff5-l77hk"
Sep 18 14:12:33.992166 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:33.992110    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b2fa854-3ea0-482b-b421-dc06a95803dc-tmp-dir\") pod \"autores-model-ae7021-kserve-7c4c8c4ff5-l77hk\" (UID: \"1b2fa854-3ea0-482b-b421-dc06a95803dc\") " pod="ai-tenant-e2e-autores-ae7021/autores-model-ae7021-kserve-7c4c8c4ff5-l77hk"
Sep 18 14:12:33.992286 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:33.992163    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b2fa854-3ea0-482b-b421-dc06a95803dc-home\") pod \"autores-model-ae7021-kserve-7c4c8c4ff5-l77hk\" (UID: \"1b2fa854-3ea0-482b-b421-dc06a95803dc\") " pod="ai-tenant-e2e-autores-ae7021/autores-model-ae7021-kserve-7c4c8c4ff5-l77hk"
Sep 18 14:12:33.992286 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:33.992200    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b2fa854-3ea0-482b-b421-dc06a95803dc-model-cache\") pod \"autores-model-ae7021-kserve-7c4c8c4ff5-l77hk\" (UID: \"1b2fa854-3ea0-482b-b421-dc06a95803dc\") " pod="ai-tenant-e2e-autores-ae7021/autores-model-ae7021-kserve-7c4c8c4ff5-l77hk"
Sep 18 14:12:33.992367 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:33.992291    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcvjh\" (UniqueName: \"kubernetes.io/projected/1b2fa854-3ea0-482b-b421-dc06a95803dc-kube-api-access-jcvjh\") pod \"autores-model-ae7021-kserve-7c4c8c4ff5-l77hk\" (UID: \"1b2fa854-3ea0-482b-b421-dc06a95803dc\") " pod="ai-tenant-e2e-autores-ae7021/autores-model-ae7021-kserve-7c4c8c4ff5-l77hk"
Sep 18 14:12:34.093305 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:34.093276    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcvjh\" (UniqueName: \"kubernetes.io/projected/1b2fa854-3ea0-482b-b421-dc06a95803dc-kube-api-access-jcvjh\") pod \"autores-model-ae7021-kserve-7c4c8c4ff5-l77hk\" (UID: \"1b2fa854-3ea0-482b-b421-dc06a95803dc\") " pod="ai-tenant-e2e-autores-ae7021/autores-model-ae7021-kserve-7c4c8c4ff5-l77hk"
Sep 18 14:12:34.093463 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:34.093346    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b2fa854-3ea0-482b-b421-dc06a95803dc-dshm\") pod \"autores-model-ae7021-kserve-7c4c8c4ff5-l77hk\" (UID: \"1b2fa854-3ea0-482b-b421-dc06a95803dc\") " pod="ai-tenant-e2e-autores-ae7021/autores-model-ae7021-kserve-7c4c8c4ff5-l77hk"
Sep 18 14:12:34.093463 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:34.093385    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b2fa854-3ea0-482b-b421-dc06a95803dc-tls-certs\") pod \"autores-model-ae7021-kserve-7c4c8c4ff5-l77hk\" (UID: \"1b2fa854-3ea0-482b-b421-dc06a95803dc\") " pod="ai-tenant-e2e-autores-ae7021/autores-model-ae7021-kserve-7c4c8c4ff5-l77hk"
Sep 18 14:12:34.093597 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:34.093478    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b2fa854-3ea0-482b-b421-dc06a95803dc-tmp-dir\") pod \"autores-model-ae7021-kserve-7c4c8c4ff5-l77hk\" (UID: \"1b2fa854-3ea0-482b-b421-dc06a95803dc\") " pod="ai-tenant-e2e-autores-ae7021/autores-model-ae7021-kserve-7c4c8c4ff5-l77hk"
Sep 18 14:12:34.093597 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:34.093518    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b2fa854-3ea0-482b-b421-dc06a95803dc-home\") pod \"autores-model-ae7021-kserve-7c4c8c4ff5-l77hk\" (UID: \"1b2fa854-3ea0-482b-b421-dc06a95803dc\") " pod="ai-tenant-e2e-autores-ae7021/autores-model-ae7021-kserve-7c4c8c4ff5-l77hk"
Sep 18 14:12:34.093597 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:34.093542    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b2fa854-3ea0-482b-b421-dc06a95803dc-model-cache\") pod \"autores-model-ae7021-kserve-7c4c8c4ff5-l77hk\" (UID: \"1b2fa854-3ea0-482b-b421-dc06a95803dc\") " pod="ai-tenant-e2e-autores-ae7021/autores-model-ae7021-kserve-7c4c8c4ff5-l77hk"
Sep 18 14:12:34.094098 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:34.094076    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b2fa854-3ea0-482b-b421-dc06a95803dc-tmp-dir\") pod \"autores-model-ae7021-kserve-7c4c8c4ff5-l77hk\" (UID: \"1b2fa854-3ea0-482b-b421-dc06a95803dc\") " pod="ai-tenant-e2e-autores-ae7021/autores-model-ae7021-kserve-7c4c8c4ff5-l77hk"
Sep 18 14:12:34.094307 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:34.094114    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b2fa854-3ea0-482b-b421-dc06a95803dc-model-cache\") pod \"autores-model-ae7021-kserve-7c4c8c4ff5-l77hk\" (UID: \"1b2fa854-3ea0-482b-b421-dc06a95803dc\") " pod="ai-tenant-e2e-autores-ae7021/autores-model-ae7021-kserve-7c4c8c4ff5-l77hk"
Sep 18 14:12:34.094436 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:34.094166    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b2fa854-3ea0-482b-b421-dc06a95803dc-home\") pod \"autores-model-ae7021-kserve-7c4c8c4ff5-l77hk\" (UID: \"1b2fa854-3ea0-482b-b421-dc06a95803dc\") " pod="ai-tenant-e2e-autores-ae7021/autores-model-ae7021-kserve-7c4c8c4ff5-l77hk"
Sep 18 14:12:34.095758 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:34.095737    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b2fa854-3ea0-482b-b421-dc06a95803dc-dshm\") pod \"autores-model-ae7021-kserve-7c4c8c4ff5-l77hk\" (UID: \"1b2fa854-3ea0-482b-b421-dc06a95803dc\") " pod="ai-tenant-e2e-autores-ae7021/autores-model-ae7021-kserve-7c4c8c4ff5-l77hk"
Sep 18 14:12:34.095985 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:34.095967    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b2fa854-3ea0-482b-b421-dc06a95803dc-tls-certs\") pod \"autores-model-ae7021-kserve-7c4c8c4ff5-l77hk\" (UID: \"1b2fa854-3ea0-482b-b421-dc06a95803dc\") " pod="ai-tenant-e2e-autores-ae7021/autores-model-ae7021-kserve-7c4c8c4ff5-l77hk"
Sep 18 14:12:34.102403 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:34.102384    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcvjh\" (UniqueName: \"kubernetes.io/projected/1b2fa854-3ea0-482b-b421-dc06a95803dc-kube-api-access-jcvjh\") pod \"autores-model-ae7021-kserve-7c4c8c4ff5-l77hk\" (UID: \"1b2fa854-3ea0-482b-b421-dc06a95803dc\") " pod="ai-tenant-e2e-autores-ae7021/autores-model-ae7021-kserve-7c4c8c4ff5-l77hk"
Sep 18 14:12:34.161470 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:34.161439    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:34.260789 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:34.260723    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-ae7021/autores-model-ae7021-kserve-7c4c8c4ff5-l77hk"
Sep 18 14:12:34.546536 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:34.546509    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-ae7021/autores-model-ae7021-kserve-7c4c8c4ff5-l77hk"]
Sep 18 14:12:34.548993 ip-10-0-143-120 kubenswrapper[2579]: W0918 14:12:34.548969    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b2fa854_3ea0_482b_b421_dc06a95803dc.slice/crio-92863dd1a26dfefe6405cc03a86946dead4ba22a836ab45b93aa43652844bf4b WatchSource:0}: Error finding container 92863dd1a26dfefe6405cc03a86946dead4ba22a836ab45b93aa43652844bf4b: Status 404 returned error can't find the container with id 92863dd1a26dfefe6405cc03a86946dead4ba22a836ab45b93aa43652844bf4b
Sep 18 14:12:35.440909 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:35.440873    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-ae7021/autores-model-ae7021-kserve-7c4c8c4ff5-l77hk" event={"ID":"1b2fa854-3ea0-482b-b421-dc06a95803dc","Type":"ContainerStarted","Data":"5405f411fd64387fe11476b50f4001060504f998f24296592858bb74cd6b4149"}
Sep 18 14:12:35.441301 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:35.440916    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-ae7021/autores-model-ae7021-kserve-7c4c8c4ff5-l77hk" event={"ID":"1b2fa854-3ea0-482b-b421-dc06a95803dc","Type":"ContainerStarted","Data":"92863dd1a26dfefe6405cc03a86946dead4ba22a836ab45b93aa43652844bf4b"}
Sep 18 14:12:35.476626 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:35.476579    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-ae7021/autores-model-ae7021-kserve-7c4c8c4ff5-l77hk" podStartSLOduration=2.476564539 podStartE2EDuration="2.476564539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:12:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:12:35.476245443 +0000 UTC m=+827.364961375" watchObservedRunningTime="2026-09-18 14:12:35.476564539 +0000 UTC m=+827.365280493"
Sep 18 14:12:37.970093 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:37.970057    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:39.218571 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:39.218538    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:40.296379 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:40.296344    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:41.244973 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:41.244944    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:42.278563 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:42.278526    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:43.161881 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:43.161849    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:44.150307 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:44.150263    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:44.261814 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:44.261780    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-ae7021/autores-model-ae7021-kserve-7c4c8c4ff5-l77hk"
Sep 18 14:12:44.261983 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:44.261828    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-ae7021/autores-model-ae7021-kserve-7c4c8c4ff5-l77hk"
Sep 18 14:12:44.274233 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:44.274207    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-ae7021/autores-model-ae7021-kserve-7c4c8c4ff5-l77hk"
Sep 18 14:12:44.494421 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:44.494353    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-ae7021/autores-model-ae7021-kserve-7c4c8c4ff5-l77hk"
Sep 18 14:12:45.156370 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:45.156338    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:45.393351 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:45.393315    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:46.172278 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:46.172247    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:47.146639 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:47.146607    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:48.645820 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:48.645781    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:51.591511 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:51.591476    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:52.173004 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:52.172957    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5f7e430d-885794f4f-nj9dn"]
Sep 18 14:12:52.173249 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:52.173199    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f7e430d-885794f4f-nj9dn" podUID="5b4b4f78-fa7b-4d03-8fee-dfea18872a95" containerName="maas-api" containerID="cri-o://deb4e162e7193326add9cfb165e88673a6e54efccfcd0a045035578e01b0e996" gracePeriod=30
Sep 18 14:12:52.209482 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:52.209458    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:52.310947 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:52.310914    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:52.412020 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:52.411987    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:52.424920 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:52.424874    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f7e430d-885794f4f-nj9dn"
Sep 18 14:12:52.516034 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:52.516004    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:52.519370 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:52.519348    2579 generic.go:356] "Generic (PLEG): container finished" podID="5b4b4f78-fa7b-4d03-8fee-dfea18872a95" containerID="deb4e162e7193326add9cfb165e88673a6e54efccfcd0a045035578e01b0e996" exitCode=0
Sep 18 14:12:52.519483 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:52.519383    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f7e430d-885794f4f-nj9dn" event={"ID":"5b4b4f78-fa7b-4d03-8fee-dfea18872a95","Type":"ContainerDied","Data":"deb4e162e7193326add9cfb165e88673a6e54efccfcd0a045035578e01b0e996"}
Sep 18 14:12:52.519483 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:52.519388    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f7e430d-885794f4f-nj9dn"
Sep 18 14:12:52.519483 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:52.519427    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f7e430d-885794f4f-nj9dn" event={"ID":"5b4b4f78-fa7b-4d03-8fee-dfea18872a95","Type":"ContainerDied","Data":"ccc1559f09a03d71f5f52789c67cfe4c6fa36c80a478d3abf0b424fe5c600ae6"}
Sep 18 14:12:52.519483 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:52.519453    2579 scope.go:117] "RemoveContainer" containerID="deb4e162e7193326add9cfb165e88673a6e54efccfcd0a045035578e01b0e996"
Sep 18 14:12:52.528019 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:52.528003    2579 scope.go:117] "RemoveContainer" containerID="deb4e162e7193326add9cfb165e88673a6e54efccfcd0a045035578e01b0e996"
Sep 18 14:12:52.528347 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:12:52.528319    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"deb4e162e7193326add9cfb165e88673a6e54efccfcd0a045035578e01b0e996\": container with ID starting with deb4e162e7193326add9cfb165e88673a6e54efccfcd0a045035578e01b0e996 not found: ID does not exist" containerID="deb4e162e7193326add9cfb165e88673a6e54efccfcd0a045035578e01b0e996"
Sep 18 14:12:52.528426 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:52.528355    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deb4e162e7193326add9cfb165e88673a6e54efccfcd0a045035578e01b0e996"} err="failed to get container status \"deb4e162e7193326add9cfb165e88673a6e54efccfcd0a045035578e01b0e996\": rpc error: code = NotFound desc = could not find container \"deb4e162e7193326add9cfb165e88673a6e54efccfcd0a045035578e01b0e996\": container with ID starting with deb4e162e7193326add9cfb165e88673a6e54efccfcd0a045035578e01b0e996 not found: ID does not exist"
Sep 18 14:12:52.552535 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:52.552512    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhcv2\" (UniqueName: \"kubernetes.io/projected/5b4b4f78-fa7b-4d03-8fee-dfea18872a95-kube-api-access-mhcv2\") pod \"5b4b4f78-fa7b-4d03-8fee-dfea18872a95\" (UID: \"5b4b4f78-fa7b-4d03-8fee-dfea18872a95\") "
Sep 18 14:12:52.552656 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:52.552553    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b4b4f78-fa7b-4d03-8fee-dfea18872a95-maas-api-tls\") pod \"5b4b4f78-fa7b-4d03-8fee-dfea18872a95\" (UID: \"5b4b4f78-fa7b-4d03-8fee-dfea18872a95\") "
Sep 18 14:12:52.552656 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:52.552603    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b4b4f78-fa7b-4d03-8fee-dfea18872a95-openshift-service-ca\") pod \"5b4b4f78-fa7b-4d03-8fee-dfea18872a95\" (UID: \"5b4b4f78-fa7b-4d03-8fee-dfea18872a95\") "
Sep 18 14:12:52.552656 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:52.552635    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b4b4f78-fa7b-4d03-8fee-dfea18872a95-metrics-certs\") pod \"5b4b4f78-fa7b-4d03-8fee-dfea18872a95\" (UID: \"5b4b4f78-fa7b-4d03-8fee-dfea18872a95\") "
Sep 18 14:12:52.553009 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:52.552986    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b4b4f78-fa7b-4d03-8fee-dfea18872a95-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5b4b4f78-fa7b-4d03-8fee-dfea18872a95" (UID: "5b4b4f78-fa7b-4d03-8fee-dfea18872a95"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:12:52.556988 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:52.556947    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b4b4f78-fa7b-4d03-8fee-dfea18872a95-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5b4b4f78-fa7b-4d03-8fee-dfea18872a95" (UID: "5b4b4f78-fa7b-4d03-8fee-dfea18872a95"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:12:52.561245 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:52.561205    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b4b4f78-fa7b-4d03-8fee-dfea18872a95-kube-api-access-mhcv2" (OuterVolumeSpecName: "kube-api-access-mhcv2") pod "5b4b4f78-fa7b-4d03-8fee-dfea18872a95" (UID: "5b4b4f78-fa7b-4d03-8fee-dfea18872a95"). InnerVolumeSpecName "kube-api-access-mhcv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:12:52.562409 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:52.562380    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b4b4f78-fa7b-4d03-8fee-dfea18872a95-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5b4b4f78-fa7b-4d03-8fee-dfea18872a95" (UID: "5b4b4f78-fa7b-4d03-8fee-dfea18872a95"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:12:52.608126 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:52.608103    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:52.653891 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:52.653867    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b4b4f78-fa7b-4d03-8fee-dfea18872a95-openshift-service-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:12:52.653891 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:52.653889    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b4b4f78-fa7b-4d03-8fee-dfea18872a95-metrics-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:12:52.654039 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:52.653900    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhcv2\" (UniqueName: \"kubernetes.io/projected/5b4b4f78-fa7b-4d03-8fee-dfea18872a95-kube-api-access-mhcv2\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:12:52.654039 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:52.653908    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b4b4f78-fa7b-4d03-8fee-dfea18872a95-maas-api-tls\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:12:52.842688 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:52.842659    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5f7e430d-885794f4f-nj9dn"]
Sep 18 14:12:52.847179 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:52.847145    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5f7e430d-885794f4f-nj9dn"]
Sep 18 14:12:52.920365 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:52.920339    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:53.132436 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:53.132358    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:53.350622 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:53.350594    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr"]
Sep 18 14:12:53.351006 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:53.350990    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b4b4f78-fa7b-4d03-8fee-dfea18872a95" containerName="maas-api"
Sep 18 14:12:53.351097 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:53.351010    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b4b4f78-fa7b-4d03-8fee-dfea18872a95" containerName="maas-api"
Sep 18 14:12:53.351097 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:53.351091    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b4b4f78-fa7b-4d03-8fee-dfea18872a95" containerName="maas-api"
Sep 18 14:12:53.355636 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:53.355611    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr"
Sep 18 14:12:53.357995 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:53.357975    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-35b94378-openshift-default-dockercfg-zq5gf\""
Sep 18 14:12:53.370700 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:53.370677    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr"]
Sep 18 14:12:53.385966 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:53.385878    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:53.460869 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:53.460834    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be0a90a3-58fa-4858-9757-664be2ddce35-istio-envoy\") pod \"e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr\" (UID: \"be0a90a3-58fa-4858-9757-664be2ddce35\") " pod="openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr"
Sep 18 14:12:53.461223 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:53.461147    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be0a90a3-58fa-4858-9757-664be2ddce35-istio-podinfo\") pod \"e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr\" (UID: \"be0a90a3-58fa-4858-9757-664be2ddce35\") " pod="openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr"
Sep 18 14:12:53.461415 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:53.461395    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmcgp\" (UniqueName: \"kubernetes.io/projected/be0a90a3-58fa-4858-9757-664be2ddce35-kube-api-access-zmcgp\") pod \"e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr\" (UID: \"be0a90a3-58fa-4858-9757-664be2ddce35\") " pod="openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr"
Sep 18 14:12:53.461621 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:53.461584    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be0a90a3-58fa-4858-9757-664be2ddce35-istiod-ca-cert\") pod \"e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr\" (UID: \"be0a90a3-58fa-4858-9757-664be2ddce35\") " pod="openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr"
Sep 18 14:12:53.461820 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:53.461800    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be0a90a3-58fa-4858-9757-664be2ddce35-istio-token\") pod \"e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr\" (UID: \"be0a90a3-58fa-4858-9757-664be2ddce35\") " pod="openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr"
Sep 18 14:12:53.461915 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:53.461889    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be0a90a3-58fa-4858-9757-664be2ddce35-istio-data\") pod \"e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr\" (UID: \"be0a90a3-58fa-4858-9757-664be2ddce35\") " pod="openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr"
Sep 18 14:12:53.461976 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:53.461935    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be0a90a3-58fa-4858-9757-664be2ddce35-workload-socket\") pod \"e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr\" (UID: \"be0a90a3-58fa-4858-9757-664be2ddce35\") " pod="openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr"
Sep 18 14:12:53.461976 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:53.461970    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be0a90a3-58fa-4858-9757-664be2ddce35-workload-certs\") pod \"e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr\" (UID: \"be0a90a3-58fa-4858-9757-664be2ddce35\") " pod="openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr"
Sep 18 14:12:53.462079 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:53.462024    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be0a90a3-58fa-4858-9757-664be2ddce35-credential-socket\") pod \"e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr\" (UID: \"be0a90a3-58fa-4858-9757-664be2ddce35\") " pod="openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr"
Sep 18 14:12:53.493344 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:53.493310    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:53.563445 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:53.563415    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be0a90a3-58fa-4858-9757-664be2ddce35-workload-socket\") pod \"e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr\" (UID: \"be0a90a3-58fa-4858-9757-664be2ddce35\") " pod="openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr"
Sep 18 14:12:53.563608 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:53.563450    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be0a90a3-58fa-4858-9757-664be2ddce35-workload-certs\") pod \"e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr\" (UID: \"be0a90a3-58fa-4858-9757-664be2ddce35\") " pod="openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr"
Sep 18 14:12:53.563608 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:53.563484    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be0a90a3-58fa-4858-9757-664be2ddce35-credential-socket\") pod \"e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr\" (UID: \"be0a90a3-58fa-4858-9757-664be2ddce35\") " pod="openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr"
Sep 18 14:12:53.563608 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:53.563506    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be0a90a3-58fa-4858-9757-664be2ddce35-istio-envoy\") pod \"e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr\" (UID: \"be0a90a3-58fa-4858-9757-664be2ddce35\") " pod="openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr"
Sep 18 14:12:53.563608 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:53.563538    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be0a90a3-58fa-4858-9757-664be2ddce35-istio-podinfo\") pod \"e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr\" (UID: \"be0a90a3-58fa-4858-9757-664be2ddce35\") " pod="openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr"
Sep 18 14:12:53.563608 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:53.563568    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmcgp\" (UniqueName: \"kubernetes.io/projected/be0a90a3-58fa-4858-9757-664be2ddce35-kube-api-access-zmcgp\") pod \"e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr\" (UID: \"be0a90a3-58fa-4858-9757-664be2ddce35\") " pod="openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr"
Sep 18 14:12:53.563608 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:53.563596    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be0a90a3-58fa-4858-9757-664be2ddce35-istiod-ca-cert\") pod \"e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr\" (UID: \"be0a90a3-58fa-4858-9757-664be2ddce35\") " pod="openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr"
Sep 18 14:12:53.563938 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:53.563748    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be0a90a3-58fa-4858-9757-664be2ddce35-istio-token\") pod \"e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr\" (UID: \"be0a90a3-58fa-4858-9757-664be2ddce35\") " pod="openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr"
Sep 18 14:12:53.563938 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:53.563856    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be0a90a3-58fa-4858-9757-664be2ddce35-istio-data\") pod \"e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr\" (UID: \"be0a90a3-58fa-4858-9757-664be2ddce35\") " pod="openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr"
Sep 18 14:12:53.563938 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:53.563900    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be0a90a3-58fa-4858-9757-664be2ddce35-workload-certs\") pod \"e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr\" (UID: \"be0a90a3-58fa-4858-9757-664be2ddce35\") " pod="openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr"
Sep 18 14:12:53.564094 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:53.563961    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be0a90a3-58fa-4858-9757-664be2ddce35-credential-socket\") pod \"e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr\" (UID: \"be0a90a3-58fa-4858-9757-664be2ddce35\") " pod="openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr"
Sep 18 14:12:53.564199 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:53.564163    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be0a90a3-58fa-4858-9757-664be2ddce35-istio-data\") pod \"e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr\" (UID: \"be0a90a3-58fa-4858-9757-664be2ddce35\") " pod="openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr"
Sep 18 14:12:53.564294 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:53.564268    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be0a90a3-58fa-4858-9757-664be2ddce35-workload-socket\") pod \"e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr\" (UID: \"be0a90a3-58fa-4858-9757-664be2ddce35\") " pod="openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr"
Sep 18 14:12:53.564538 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:53.564517    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be0a90a3-58fa-4858-9757-664be2ddce35-istiod-ca-cert\") pod \"e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr\" (UID: \"be0a90a3-58fa-4858-9757-664be2ddce35\") " pod="openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr"
Sep 18 14:12:53.565996 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:53.565977    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be0a90a3-58fa-4858-9757-664be2ddce35-istio-podinfo\") pod \"e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr\" (UID: \"be0a90a3-58fa-4858-9757-664be2ddce35\") " pod="openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr"
Sep 18 14:12:53.566093 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:53.566076    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be0a90a3-58fa-4858-9757-664be2ddce35-istio-envoy\") pod \"e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr\" (UID: \"be0a90a3-58fa-4858-9757-664be2ddce35\") " pod="openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr"
Sep 18 14:12:53.572570 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:53.572548    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmcgp\" (UniqueName: \"kubernetes.io/projected/be0a90a3-58fa-4858-9757-664be2ddce35-kube-api-access-zmcgp\") pod \"e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr\" (UID: \"be0a90a3-58fa-4858-9757-664be2ddce35\") " pod="openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr"
Sep 18 14:12:53.572946 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:53.572923    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be0a90a3-58fa-4858-9757-664be2ddce35-istio-token\") pod \"e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr\" (UID: \"be0a90a3-58fa-4858-9757-664be2ddce35\") " pod="openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr"
Sep 18 14:12:53.665922 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:53.665868    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr"
Sep 18 14:12:53.836634 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:53.836602    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:53.944384 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:53.944362    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr"]
Sep 18 14:12:53.946131 ip-10-0-143-120 kubenswrapper[2579]: W0918 14:12:53.946103    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe0a90a3_58fa_4858_9757_664be2ddce35.slice/crio-f30209feb530660e1354cef4e5d5196afdae867f175ad161dae64c43eda4f12a WatchSource:0}: Error finding container f30209feb530660e1354cef4e5d5196afdae867f175ad161dae64c43eda4f12a: Status 404 returned error can't find the container with id f30209feb530660e1354cef4e5d5196afdae867f175ad161dae64c43eda4f12a
Sep 18 14:12:53.948640 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:53.948606    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:12:53.948702 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:53.948673    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:12:53.948749 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:53.948705    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:12:54.036415 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:54.036383    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b4b4f78-fa7b-4d03-8fee-dfea18872a95" path="/var/lib/kubelet/pods/5b4b4f78-fa7b-4d03-8fee-dfea18872a95/volumes"
Sep 18 14:12:54.531085 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:54.531048    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr" event={"ID":"be0a90a3-58fa-4858-9757-664be2ddce35","Type":"ContainerStarted","Data":"4e9aaa2bf1fda619b10c720d4d8e33091d41b59daeb3b696084a8bb7a96c1305"}
Sep 18 14:12:54.531309 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:54.531111    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr" event={"ID":"be0a90a3-58fa-4858-9757-664be2ddce35","Type":"ContainerStarted","Data":"f30209feb530660e1354cef4e5d5196afdae867f175ad161dae64c43eda4f12a"}
Sep 18 14:12:54.558974 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:54.558912    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr" podStartSLOduration=1.55889251 podStartE2EDuration="1.55889251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:12:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:12:54.555974807 +0000 UTC m=+846.444690750" watchObservedRunningTime="2026-09-18 14:12:54.55889251 +0000 UTC m=+846.447608441"
Sep 18 14:12:54.666829 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:54.666787    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr"
Sep 18 14:12:54.672764 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:54.672723    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr"
Sep 18 14:12:54.889568 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:54.889526    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:55.536139 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:55.536108    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr"
Sep 18 14:12:55.536996 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:55.536977    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr"
Sep 18 14:12:55.747866 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:55.747834    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:56.066084 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:56.066045    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:56.180599 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:56.180560    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:56.757427 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:56.757389    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:57.537500 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:57.537467    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:58.382022 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:58.381989    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-35b94378-6cc7f7bcdc-b9jj2"]
Sep 18 14:12:58.385780 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:58.385759    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35b94378-6cc7f7bcdc-b9jj2"
Sep 18 14:12:58.391430 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:58.391406    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-35b94378-serving-cert\""
Sep 18 14:12:58.404797 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:58.404771    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-35b94378-6cc7f7bcdc-b9jj2"]
Sep 18 14:12:58.422706 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:58.422684    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:58.506478 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:58.506452    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49c5b322-39c0-43cc-ac51-6adb4f89b9d8-maas-api-tls\") pod \"maas-api-e2e-ait-35b94378-6cc7f7bcdc-b9jj2\" (UID: \"49c5b322-39c0-43cc-ac51-6adb4f89b9d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35b94378-6cc7f7bcdc-b9jj2"
Sep 18 14:12:58.506630 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:58.506484    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49c5b322-39c0-43cc-ac51-6adb4f89b9d8-openshift-service-ca\") pod \"maas-api-e2e-ait-35b94378-6cc7f7bcdc-b9jj2\" (UID: \"49c5b322-39c0-43cc-ac51-6adb4f89b9d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35b94378-6cc7f7bcdc-b9jj2"
Sep 18 14:12:58.506630 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:58.506523    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwzp5\" (UniqueName: \"kubernetes.io/projected/49c5b322-39c0-43cc-ac51-6adb4f89b9d8-kube-api-access-lwzp5\") pod \"maas-api-e2e-ait-35b94378-6cc7f7bcdc-b9jj2\" (UID: \"49c5b322-39c0-43cc-ac51-6adb4f89b9d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35b94378-6cc7f7bcdc-b9jj2"
Sep 18 14:12:58.506630 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:58.506607    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49c5b322-39c0-43cc-ac51-6adb4f89b9d8-metrics-certs\") pod \"maas-api-e2e-ait-35b94378-6cc7f7bcdc-b9jj2\" (UID: \"49c5b322-39c0-43cc-ac51-6adb4f89b9d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35b94378-6cc7f7bcdc-b9jj2"
Sep 18 14:12:58.521774 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:58.521751    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:58.607845 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:58.607821    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwzp5\" (UniqueName: \"kubernetes.io/projected/49c5b322-39c0-43cc-ac51-6adb4f89b9d8-kube-api-access-lwzp5\") pod \"maas-api-e2e-ait-35b94378-6cc7f7bcdc-b9jj2\" (UID: \"49c5b322-39c0-43cc-ac51-6adb4f89b9d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35b94378-6cc7f7bcdc-b9jj2"
Sep 18 14:12:58.607978 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:58.607877    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49c5b322-39c0-43cc-ac51-6adb4f89b9d8-metrics-certs\") pod \"maas-api-e2e-ait-35b94378-6cc7f7bcdc-b9jj2\" (UID: \"49c5b322-39c0-43cc-ac51-6adb4f89b9d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35b94378-6cc7f7bcdc-b9jj2"
Sep 18 14:12:58.607978 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:58.607925    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49c5b322-39c0-43cc-ac51-6adb4f89b9d8-maas-api-tls\") pod \"maas-api-e2e-ait-35b94378-6cc7f7bcdc-b9jj2\" (UID: \"49c5b322-39c0-43cc-ac51-6adb4f89b9d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35b94378-6cc7f7bcdc-b9jj2"
Sep 18 14:12:58.607978 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:58.607939    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49c5b322-39c0-43cc-ac51-6adb4f89b9d8-openshift-service-ca\") pod \"maas-api-e2e-ait-35b94378-6cc7f7bcdc-b9jj2\" (UID: \"49c5b322-39c0-43cc-ac51-6adb4f89b9d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35b94378-6cc7f7bcdc-b9jj2"
Sep 18 14:12:58.608793 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:58.608772    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49c5b322-39c0-43cc-ac51-6adb4f89b9d8-openshift-service-ca\") pod \"maas-api-e2e-ait-35b94378-6cc7f7bcdc-b9jj2\" (UID: \"49c5b322-39c0-43cc-ac51-6adb4f89b9d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35b94378-6cc7f7bcdc-b9jj2"
Sep 18 14:12:58.610444 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:58.610418    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49c5b322-39c0-43cc-ac51-6adb4f89b9d8-metrics-certs\") pod \"maas-api-e2e-ait-35b94378-6cc7f7bcdc-b9jj2\" (UID: \"49c5b322-39c0-43cc-ac51-6adb4f89b9d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35b94378-6cc7f7bcdc-b9jj2"
Sep 18 14:12:58.610541 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:58.610465    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49c5b322-39c0-43cc-ac51-6adb4f89b9d8-maas-api-tls\") pod \"maas-api-e2e-ait-35b94378-6cc7f7bcdc-b9jj2\" (UID: \"49c5b322-39c0-43cc-ac51-6adb4f89b9d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35b94378-6cc7f7bcdc-b9jj2"
Sep 18 14:12:58.617052 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:58.617027    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwzp5\" (UniqueName: \"kubernetes.io/projected/49c5b322-39c0-43cc-ac51-6adb4f89b9d8-kube-api-access-lwzp5\") pod \"maas-api-e2e-ait-35b94378-6cc7f7bcdc-b9jj2\" (UID: \"49c5b322-39c0-43cc-ac51-6adb4f89b9d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35b94378-6cc7f7bcdc-b9jj2"
Sep 18 14:12:58.620743 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:58.620719    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:58.696834 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:58.696783    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35b94378-6cc7f7bcdc-b9jj2"
Sep 18 14:12:58.946498 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:58.946466    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:58.976193 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:58.976152    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-35b94378-6cc7f7bcdc-b9jj2"]
Sep 18 14:12:58.978025 ip-10-0-143-120 kubenswrapper[2579]: W0918 14:12:58.977997    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49c5b322_39c0_43cc_ac51_6adb4f89b9d8.slice/crio-2997d5ef877583575b4f7cb13a1b19e852621db3bc0339d1b5822e2ae5335b19 WatchSource:0}: Error finding container 2997d5ef877583575b4f7cb13a1b19e852621db3bc0339d1b5822e2ae5335b19: Status 404 returned error can't find the container with id 2997d5ef877583575b4f7cb13a1b19e852621db3bc0339d1b5822e2ae5335b19
Sep 18 14:12:59.053161 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:59.053137    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:12:59.555922 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:59.555813    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35b94378-6cc7f7bcdc-b9jj2" event={"ID":"49c5b322-39c0-43cc-ac51-6adb4f89b9d8","Type":"ContainerStarted","Data":"c1adefa98280762e5e1fe65076b51cc49daef2d31c6898c8b592fddb5595c8fa"}
Sep 18 14:12:59.555922 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:59.555874    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35b94378-6cc7f7bcdc-b9jj2" event={"ID":"49c5b322-39c0-43cc-ac51-6adb4f89b9d8","Type":"ContainerStarted","Data":"2997d5ef877583575b4f7cb13a1b19e852621db3bc0339d1b5822e2ae5335b19"}
Sep 18 14:12:59.556386 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:59.555984    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35b94378-6cc7f7bcdc-b9jj2"
Sep 18 14:12:59.590215 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:59.590155    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35b94378-6cc7f7bcdc-b9jj2" podStartSLOduration=1.082761969 podStartE2EDuration="1.590139196s" totalImagesPullingTime="507.377227ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:12:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:12:59.587434665 +0000 UTC m=+851.476150596" watchObservedRunningTime="2026-09-18 14:12:59.590139196 +0000 UTC m=+851.478855127"
Sep 18 14:12:59.674349 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:12:59.674323    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:00.458369 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:00.458327    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:01.329399 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:01.329367    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:02.043713 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:02.043679    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:02.732286 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:02.732253    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:05.566510 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:05.566480    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35b94378-6cc7f7bcdc-b9jj2"
Sep 18 14:13:06.054999 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:06.054957    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:08.246665 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:08.246631    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:09.152243 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:09.152213    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:10.041320 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:10.041287    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:10.930989 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:10.930954    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:11.648592 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:11.648539    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:12.540109 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:12.540068    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:13.326921 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:13.326871    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:14.116545 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:14.116511    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:15.008259 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:15.008222    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:15.940713 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:15.940647    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:16.638151 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:16.638104    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:17.435903 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:17.435872    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:18.118037 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:18.118004    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:18.942165 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:18.942129    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:19.495888 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:19.495852    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-ae7021-ccb785d8c-ztcqg"]
Sep 18 14:13:19.496378 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:19.496092    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ae7021-ccb785d8c-ztcqg" podUID="0bee7612-56a4-4f90-a62a-8a7674f5634d" containerName="maas-api" containerID="cri-o://78f22e5c9aa3a88dd975629be474aaf3e4f026833fa0aa7b8d3c31da044b2efa" gracePeriod=30
Sep 18 14:13:19.524898 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:19.524836    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:19.604983 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:19.604951    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:19.651764 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:19.651664    2579 generic.go:356] "Generic (PLEG): container finished" podID="0bee7612-56a4-4f90-a62a-8a7674f5634d" containerID="78f22e5c9aa3a88dd975629be474aaf3e4f026833fa0aa7b8d3c31da044b2efa" exitCode=0
Sep 18 14:13:19.651764 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:19.651707    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ae7021-ccb785d8c-ztcqg" event={"ID":"0bee7612-56a4-4f90-a62a-8a7674f5634d","Type":"ContainerDied","Data":"78f22e5c9aa3a88dd975629be474aaf3e4f026833fa0aa7b8d3c31da044b2efa"}
Sep 18 14:13:19.702531 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:19.702495    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:19.776207 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:19.775991    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:19.811239 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:19.810142    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ae7021-ccb785d8c-ztcqg"
Sep 18 14:13:19.870722 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:19.870636    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:19.950962 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:19.950930    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:20.000019 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:19.999986    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bee7612-56a4-4f90-a62a-8a7674f5634d-openshift-service-ca\") pod \"0bee7612-56a4-4f90-a62a-8a7674f5634d\" (UID: \"0bee7612-56a4-4f90-a62a-8a7674f5634d\") "
Sep 18 14:13:20.000210 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:20.000044    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bee7612-56a4-4f90-a62a-8a7674f5634d-maas-api-tls\") pod \"0bee7612-56a4-4f90-a62a-8a7674f5634d\" (UID: \"0bee7612-56a4-4f90-a62a-8a7674f5634d\") "
Sep 18 14:13:20.000210 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:20.000070    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4z6ns\" (UniqueName: \"kubernetes.io/projected/0bee7612-56a4-4f90-a62a-8a7674f5634d-kube-api-access-4z6ns\") pod \"0bee7612-56a4-4f90-a62a-8a7674f5634d\" (UID: \"0bee7612-56a4-4f90-a62a-8a7674f5634d\") "
Sep 18 14:13:20.000210 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:20.000137    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bee7612-56a4-4f90-a62a-8a7674f5634d-metrics-certs\") pod \"0bee7612-56a4-4f90-a62a-8a7674f5634d\" (UID: \"0bee7612-56a4-4f90-a62a-8a7674f5634d\") "
Sep 18 14:13:20.000409 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:20.000318    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bee7612-56a4-4f90-a62a-8a7674f5634d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0bee7612-56a4-4f90-a62a-8a7674f5634d" (UID: "0bee7612-56a4-4f90-a62a-8a7674f5634d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:13:20.002247 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:20.002214    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bee7612-56a4-4f90-a62a-8a7674f5634d-kube-api-access-4z6ns" (OuterVolumeSpecName: "kube-api-access-4z6ns") pod "0bee7612-56a4-4f90-a62a-8a7674f5634d" (UID: "0bee7612-56a4-4f90-a62a-8a7674f5634d"). InnerVolumeSpecName "kube-api-access-4z6ns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:13:20.002358 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:20.002270    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bee7612-56a4-4f90-a62a-8a7674f5634d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0bee7612-56a4-4f90-a62a-8a7674f5634d" (UID: "0bee7612-56a4-4f90-a62a-8a7674f5634d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:13:20.002429 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:20.002413    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bee7612-56a4-4f90-a62a-8a7674f5634d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0bee7612-56a4-4f90-a62a-8a7674f5634d" (UID: "0bee7612-56a4-4f90-a62a-8a7674f5634d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:13:20.100736 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:20.100710    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bee7612-56a4-4f90-a62a-8a7674f5634d-openshift-service-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:13:20.100736 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:20.100729    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bee7612-56a4-4f90-a62a-8a7674f5634d-maas-api-tls\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:13:20.100736 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:20.100739    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z6ns\" (UniqueName: \"kubernetes.io/projected/0bee7612-56a4-4f90-a62a-8a7674f5634d-kube-api-access-4z6ns\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:13:20.100923 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:20.100747    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bee7612-56a4-4f90-a62a-8a7674f5634d-metrics-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:13:20.160351 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:20.160312    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:20.659664 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:20.659627    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ae7021-ccb785d8c-ztcqg" event={"ID":"0bee7612-56a4-4f90-a62a-8a7674f5634d","Type":"ContainerDied","Data":"b0879fbf10f30f039eef6439125c5400211e453d616f8cb87f32b15a88e1c010"}
Sep 18 14:13:20.660124 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:20.659708    2579 scope.go:117] "RemoveContainer" containerID="78f22e5c9aa3a88dd975629be474aaf3e4f026833fa0aa7b8d3c31da044b2efa"
Sep 18 14:13:20.660124 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:20.659642    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ae7021-ccb785d8c-ztcqg"
Sep 18 14:13:20.661500 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:20.661450    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:20.677927 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:20.677895    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-ae7021-ccb785d8c-ztcqg"]
Sep 18 14:13:20.682429 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:20.682399    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-ae7021-ccb785d8c-ztcqg"]
Sep 18 14:13:21.938381 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:21.938343    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:22.036354 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:22.036321    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bee7612-56a4-4f90-a62a-8a7674f5634d" path="/var/lib/kubelet/pods/0bee7612-56a4-4f90-a62a-8a7674f5634d/volumes"
Sep 18 14:13:23.143002 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:23.142964    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:23.675347 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:23.675291    2579 generic.go:356] "Generic (PLEG): container finished" podID="be0a90a3-58fa-4858-9757-664be2ddce35" containerID="4e9aaa2bf1fda619b10c720d4d8e33091d41b59daeb3b696084a8bb7a96c1305" exitCode=0
Sep 18 14:13:23.675487 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:23.675361    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr" event={"ID":"be0a90a3-58fa-4858-9757-664be2ddce35","Type":"ContainerDied","Data":"4e9aaa2bf1fda619b10c720d4d8e33091d41b59daeb3b696084a8bb7a96c1305"}
Sep 18 14:13:23.675739 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:23.675715    2579 scope.go:117] "RemoveContainer" containerID="4e9aaa2bf1fda619b10c720d4d8e33091d41b59daeb3b696084a8bb7a96c1305"
Sep 18 14:13:23.676771 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:23.676744    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:13:23.676828 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:23.676815    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:13:23.676875 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:23.676853    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:13:23.952207 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:23.952149    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:24.681855 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:24.681824    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr" event={"ID":"be0a90a3-58fa-4858-9757-664be2ddce35","Type":"ContainerStarted","Data":"a53ca2e6e9eaddcdd0e5c7ca452d4b7bf5dcb28599152535d66c69d86a0511d0"}
Sep 18 14:13:24.742313 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:24.742288    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:25.135875 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:25.135838    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-ae7021/autores-model-ae7021-kserve-7c4c8c4ff5-l77hk"]
Sep 18 14:13:25.245252 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:25.245219    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:25.536946 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:25.536842    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr"
Sep 18 14:13:25.664201 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:25.664154    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:25.666779 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:25.666757    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr"
Sep 18 14:13:25.668106 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:25.668068    2579 patch_prober.go:28] interesting pod/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Sep 18 14:13:25.668243 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:25.668126    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr" podUID="be0a90a3-58fa-4858-9757-664be2ddce35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Sep 18 14:13:26.667459 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:26.667018    2579 patch_prober.go:28] interesting pod/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Sep 18 14:13:26.667459 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:26.667077    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr" podUID="be0a90a3-58fa-4858-9757-664be2ddce35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Sep 18 14:13:27.052111 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:27.052068    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-35b94378-6cc7f7bcdc-b9jj2"]
Sep 18 14:13:27.052396 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:27.052367    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35b94378-6cc7f7bcdc-b9jj2" podUID="49c5b322-39c0-43cc-ac51-6adb4f89b9d8" containerName="maas-api" containerID="cri-o://c1adefa98280762e5e1fe65076b51cc49daef2d31c6898c8b592fddb5595c8fa" gracePeriod=30
Sep 18 14:13:27.084389 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:27.084357    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:27.176245 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:27.176207    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:27.249652 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:27.249619    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:27.331557 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:27.331471    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:27.352942 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:27.352919    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35b94378-6cc7f7bcdc-b9jj2"
Sep 18 14:13:27.374702 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:27.373839    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49c5b322-39c0-43cc-ac51-6adb4f89b9d8-openshift-service-ca\") pod \"49c5b322-39c0-43cc-ac51-6adb4f89b9d8\" (UID: \"49c5b322-39c0-43cc-ac51-6adb4f89b9d8\") "
Sep 18 14:13:27.374702 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:27.373889    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lwzp5\" (UniqueName: \"kubernetes.io/projected/49c5b322-39c0-43cc-ac51-6adb4f89b9d8-kube-api-access-lwzp5\") pod \"49c5b322-39c0-43cc-ac51-6adb4f89b9d8\" (UID: \"49c5b322-39c0-43cc-ac51-6adb4f89b9d8\") "
Sep 18 14:13:27.374702 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:27.374022    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49c5b322-39c0-43cc-ac51-6adb4f89b9d8-metrics-certs\") pod \"49c5b322-39c0-43cc-ac51-6adb4f89b9d8\" (UID: \"49c5b322-39c0-43cc-ac51-6adb4f89b9d8\") "
Sep 18 14:13:27.374702 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:27.374053    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49c5b322-39c0-43cc-ac51-6adb4f89b9d8-maas-api-tls\") pod \"49c5b322-39c0-43cc-ac51-6adb4f89b9d8\" (UID: \"49c5b322-39c0-43cc-ac51-6adb4f89b9d8\") "
Sep 18 14:13:27.376076 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:27.376027    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49c5b322-39c0-43cc-ac51-6adb4f89b9d8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "49c5b322-39c0-43cc-ac51-6adb4f89b9d8" (UID: "49c5b322-39c0-43cc-ac51-6adb4f89b9d8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:13:27.388463 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:27.388400    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49c5b322-39c0-43cc-ac51-6adb4f89b9d8-kube-api-access-lwzp5" (OuterVolumeSpecName: "kube-api-access-lwzp5") pod "49c5b322-39c0-43cc-ac51-6adb4f89b9d8" (UID: "49c5b322-39c0-43cc-ac51-6adb4f89b9d8"). InnerVolumeSpecName "kube-api-access-lwzp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:13:27.390493 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:27.388900    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49c5b322-39c0-43cc-ac51-6adb4f89b9d8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "49c5b322-39c0-43cc-ac51-6adb4f89b9d8" (UID: "49c5b322-39c0-43cc-ac51-6adb4f89b9d8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:13:27.390493 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:27.390285    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49c5b322-39c0-43cc-ac51-6adb4f89b9d8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "49c5b322-39c0-43cc-ac51-6adb4f89b9d8" (UID: "49c5b322-39c0-43cc-ac51-6adb4f89b9d8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:13:27.477895 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:27.477650    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49c5b322-39c0-43cc-ac51-6adb4f89b9d8-metrics-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:13:27.477895 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:27.477685    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49c5b322-39c0-43cc-ac51-6adb4f89b9d8-maas-api-tls\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:13:27.477895 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:27.477703    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49c5b322-39c0-43cc-ac51-6adb4f89b9d8-openshift-service-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:13:27.477895 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:27.477718    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwzp5\" (UniqueName: \"kubernetes.io/projected/49c5b322-39c0-43cc-ac51-6adb4f89b9d8-kube-api-access-lwzp5\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:13:27.667028 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:27.666532    2579 patch_prober.go:28] interesting pod/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Sep 18 14:13:27.667028 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:27.666598    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr" podUID="be0a90a3-58fa-4858-9757-664be2ddce35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Sep 18 14:13:27.697538 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:27.697501    2579 generic.go:356] "Generic (PLEG): container finished" podID="49c5b322-39c0-43cc-ac51-6adb4f89b9d8" containerID="c1adefa98280762e5e1fe65076b51cc49daef2d31c6898c8b592fddb5595c8fa" exitCode=0
Sep 18 14:13:27.697923 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:27.697551    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35b94378-6cc7f7bcdc-b9jj2"
Sep 18 14:13:27.697923 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:27.697588    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35b94378-6cc7f7bcdc-b9jj2" event={"ID":"49c5b322-39c0-43cc-ac51-6adb4f89b9d8","Type":"ContainerDied","Data":"c1adefa98280762e5e1fe65076b51cc49daef2d31c6898c8b592fddb5595c8fa"}
Sep 18 14:13:27.697923 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:27.697644    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35b94378-6cc7f7bcdc-b9jj2" event={"ID":"49c5b322-39c0-43cc-ac51-6adb4f89b9d8","Type":"ContainerDied","Data":"2997d5ef877583575b4f7cb13a1b19e852621db3bc0339d1b5822e2ae5335b19"}
Sep 18 14:13:27.697923 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:27.697667    2579 scope.go:117] "RemoveContainer" containerID="c1adefa98280762e5e1fe65076b51cc49daef2d31c6898c8b592fddb5595c8fa"
Sep 18 14:13:27.706114 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:27.706098    2579 scope.go:117] "RemoveContainer" containerID="c1adefa98280762e5e1fe65076b51cc49daef2d31c6898c8b592fddb5595c8fa"
Sep 18 14:13:27.706388 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:13:27.706371    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1adefa98280762e5e1fe65076b51cc49daef2d31c6898c8b592fddb5595c8fa\": container with ID starting with c1adefa98280762e5e1fe65076b51cc49daef2d31c6898c8b592fddb5595c8fa not found: ID does not exist" containerID="c1adefa98280762e5e1fe65076b51cc49daef2d31c6898c8b592fddb5595c8fa"
Sep 18 14:13:27.706451 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:27.706396    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1adefa98280762e5e1fe65076b51cc49daef2d31c6898c8b592fddb5595c8fa"} err="failed to get container status \"c1adefa98280762e5e1fe65076b51cc49daef2d31c6898c8b592fddb5595c8fa\": rpc error: code = NotFound desc = could not find container \"c1adefa98280762e5e1fe65076b51cc49daef2d31c6898c8b592fddb5595c8fa\": container with ID starting with c1adefa98280762e5e1fe65076b51cc49daef2d31c6898c8b592fddb5595c8fa not found: ID does not exist"
Sep 18 14:13:27.720552 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:27.720530    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-35b94378-6cc7f7bcdc-b9jj2"]
Sep 18 14:13:27.723950 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:27.723929    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-35b94378-6cc7f7bcdc-b9jj2"]
Sep 18 14:13:27.845559 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:27.845527    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:28.035778 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:28.035683    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49c5b322-39c0-43cc-ac51-6adb4f89b9d8" path="/var/lib/kubelet/pods/49c5b322-39c0-43cc-ac51-6adb4f89b9d8/volumes"
Sep 18 14:13:28.422917 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:28.422882    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr"]
Sep 18 14:13:28.423092 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:28.423064    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr" podUID="be0a90a3-58fa-4858-9757-664be2ddce35" containerName="istio-proxy" containerID="cri-o://a53ca2e6e9eaddcdd0e5c7ca452d4b7bf5dcb28599152535d66c69d86a0511d0" gracePeriod=30
Sep 18 14:13:28.449627 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:28.449603    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:28.625248 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:28.625217    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:28.872285 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:28.872256    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:29.184894 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:29.184802    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:29.424426 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:29.424395    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:29.526655 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:29.526577    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:30.599621 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:30.599583    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:31.480494 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:31.480458    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:32.071037 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:32.071003    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:32.152114 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:32.152090    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:32.399090 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:32.399014    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:33.120617 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:33.120590    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:33.843347 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:33.843319    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:34.701378 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:34.701347    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:35.389505 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:35.389474    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a721b5af-77c55bcbf6-th7wp"]
Sep 18 14:13:35.389882 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:35.389865    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49c5b322-39c0-43cc-ac51-6adb4f89b9d8" containerName="maas-api"
Sep 18 14:13:35.389974 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:35.389885    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="49c5b322-39c0-43cc-ac51-6adb4f89b9d8" containerName="maas-api"
Sep 18 14:13:35.389974 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:35.389924    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bee7612-56a4-4f90-a62a-8a7674f5634d" containerName="maas-api"
Sep 18 14:13:35.389974 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:35.389933    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bee7612-56a4-4f90-a62a-8a7674f5634d" containerName="maas-api"
Sep 18 14:13:35.390127 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:35.390014    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="49c5b322-39c0-43cc-ac51-6adb4f89b9d8" containerName="maas-api"
Sep 18 14:13:35.390127 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:35.390030    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bee7612-56a4-4f90-a62a-8a7674f5634d" containerName="maas-api"
Sep 18 14:13:35.394481 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:35.394459    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a721b5af-77c55bcbf6-th7wp"
Sep 18 14:13:35.396629 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:35.396608    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-a721b5af-serving-cert\""
Sep 18 14:13:35.406669 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:35.406649    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a721b5af-77c55bcbf6-th7wp"]
Sep 18 14:13:35.427041 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:35.427019    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:35.446367 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:35.446344    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fb515b9-6a12-4753-91d0-77bb90af103e-maas-api-tls\") pod \"maas-api-e2e-derive-a721b5af-77c55bcbf6-th7wp\" (UID: \"8fb515b9-6a12-4753-91d0-77bb90af103e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a721b5af-77c55bcbf6-th7wp"
Sep 18 14:13:35.446462 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:35.446409    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fb515b9-6a12-4753-91d0-77bb90af103e-openshift-service-ca\") pod \"maas-api-e2e-derive-a721b5af-77c55bcbf6-th7wp\" (UID: \"8fb515b9-6a12-4753-91d0-77bb90af103e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a721b5af-77c55bcbf6-th7wp"
Sep 18 14:13:35.446462 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:35.446437    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fb515b9-6a12-4753-91d0-77bb90af103e-metrics-certs\") pod \"maas-api-e2e-derive-a721b5af-77c55bcbf6-th7wp\" (UID: \"8fb515b9-6a12-4753-91d0-77bb90af103e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a721b5af-77c55bcbf6-th7wp"
Sep 18 14:13:35.446462 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:35.446458    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfjgw\" (UniqueName: \"kubernetes.io/projected/8fb515b9-6a12-4753-91d0-77bb90af103e-kube-api-access-gfjgw\") pod \"maas-api-e2e-derive-a721b5af-77c55bcbf6-th7wp\" (UID: \"8fb515b9-6a12-4753-91d0-77bb90af103e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a721b5af-77c55bcbf6-th7wp"
Sep 18 14:13:35.509071 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:35.509025    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:35.547423 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:35.547395    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fb515b9-6a12-4753-91d0-77bb90af103e-maas-api-tls\") pod \"maas-api-e2e-derive-a721b5af-77c55bcbf6-th7wp\" (UID: \"8fb515b9-6a12-4753-91d0-77bb90af103e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a721b5af-77c55bcbf6-th7wp"
Sep 18 14:13:35.547594 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:35.547461    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fb515b9-6a12-4753-91d0-77bb90af103e-openshift-service-ca\") pod \"maas-api-e2e-derive-a721b5af-77c55bcbf6-th7wp\" (UID: \"8fb515b9-6a12-4753-91d0-77bb90af103e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a721b5af-77c55bcbf6-th7wp"
Sep 18 14:13:35.547594 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:35.547477    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fb515b9-6a12-4753-91d0-77bb90af103e-metrics-certs\") pod \"maas-api-e2e-derive-a721b5af-77c55bcbf6-th7wp\" (UID: \"8fb515b9-6a12-4753-91d0-77bb90af103e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a721b5af-77c55bcbf6-th7wp"
Sep 18 14:13:35.547594 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:35.547501    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfjgw\" (UniqueName: \"kubernetes.io/projected/8fb515b9-6a12-4753-91d0-77bb90af103e-kube-api-access-gfjgw\") pod \"maas-api-e2e-derive-a721b5af-77c55bcbf6-th7wp\" (UID: \"8fb515b9-6a12-4753-91d0-77bb90af103e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a721b5af-77c55bcbf6-th7wp"
Sep 18 14:13:35.547594 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:13:35.547541    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-a721b5af-serving-cert: secret "maas-api-e2e-derive-a721b5af-serving-cert" not found
Sep 18 14:13:35.547835 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:13:35.547600    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fb515b9-6a12-4753-91d0-77bb90af103e-maas-api-tls podName:8fb515b9-6a12-4753-91d0-77bb90af103e nodeName:}" failed. No retries permitted until 2026-09-18 14:13:36.047584037 +0000 UTC m=+887.936299946 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8fb515b9-6a12-4753-91d0-77bb90af103e-maas-api-tls") pod "maas-api-e2e-derive-a721b5af-77c55bcbf6-th7wp" (UID: "8fb515b9-6a12-4753-91d0-77bb90af103e") : secret "maas-api-e2e-derive-a721b5af-serving-cert" not found
Sep 18 14:13:35.548142 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:35.548118    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fb515b9-6a12-4753-91d0-77bb90af103e-openshift-service-ca\") pod \"maas-api-e2e-derive-a721b5af-77c55bcbf6-th7wp\" (UID: \"8fb515b9-6a12-4753-91d0-77bb90af103e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a721b5af-77c55bcbf6-th7wp"
Sep 18 14:13:35.549906 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:35.549885    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fb515b9-6a12-4753-91d0-77bb90af103e-metrics-certs\") pod \"maas-api-e2e-derive-a721b5af-77c55bcbf6-th7wp\" (UID: \"8fb515b9-6a12-4753-91d0-77bb90af103e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a721b5af-77c55bcbf6-th7wp"
Sep 18 14:13:35.556030 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:35.556005    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfjgw\" (UniqueName: \"kubernetes.io/projected/8fb515b9-6a12-4753-91d0-77bb90af103e-kube-api-access-gfjgw\") pod \"maas-api-e2e-derive-a721b5af-77c55bcbf6-th7wp\" (UID: \"8fb515b9-6a12-4753-91d0-77bb90af103e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a721b5af-77c55bcbf6-th7wp"
Sep 18 14:13:35.935210 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:35.935158    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:35.989955 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:35.989932    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr"
Sep 18 14:13:36.019546 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:36.019488    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:36.050269 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:36.050238    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be0a90a3-58fa-4858-9757-664be2ddce35-istio-token\") pod \"be0a90a3-58fa-4858-9757-664be2ddce35\" (UID: \"be0a90a3-58fa-4858-9757-664be2ddce35\") "
Sep 18 14:13:36.050418 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:36.050289    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be0a90a3-58fa-4858-9757-664be2ddce35-workload-certs\") pod \"be0a90a3-58fa-4858-9757-664be2ddce35\" (UID: \"be0a90a3-58fa-4858-9757-664be2ddce35\") "
Sep 18 14:13:36.050418 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:36.050330    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be0a90a3-58fa-4858-9757-664be2ddce35-istio-data\") pod \"be0a90a3-58fa-4858-9757-664be2ddce35\" (UID: \"be0a90a3-58fa-4858-9757-664be2ddce35\") "
Sep 18 14:13:36.050418 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:36.050369    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmcgp\" (UniqueName: \"kubernetes.io/projected/be0a90a3-58fa-4858-9757-664be2ddce35-kube-api-access-zmcgp\") pod \"be0a90a3-58fa-4858-9757-664be2ddce35\" (UID: \"be0a90a3-58fa-4858-9757-664be2ddce35\") "
Sep 18 14:13:36.050591 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:36.050426    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be0a90a3-58fa-4858-9757-664be2ddce35-istio-envoy\") pod \"be0a90a3-58fa-4858-9757-664be2ddce35\" (UID: \"be0a90a3-58fa-4858-9757-664be2ddce35\") "
Sep 18 14:13:36.050591 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:36.050454    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be0a90a3-58fa-4858-9757-664be2ddce35-workload-socket\") pod \"be0a90a3-58fa-4858-9757-664be2ddce35\" (UID: \"be0a90a3-58fa-4858-9757-664be2ddce35\") "
Sep 18 14:13:36.050591 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:36.050483    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be0a90a3-58fa-4858-9757-664be2ddce35-istiod-ca-cert\") pod \"be0a90a3-58fa-4858-9757-664be2ddce35\" (UID: \"be0a90a3-58fa-4858-9757-664be2ddce35\") "
Sep 18 14:13:36.050591 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:36.050538    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be0a90a3-58fa-4858-9757-664be2ddce35-credential-socket\") pod \"be0a90a3-58fa-4858-9757-664be2ddce35\" (UID: \"be0a90a3-58fa-4858-9757-664be2ddce35\") "
Sep 18 14:13:36.050591 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:36.050583    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be0a90a3-58fa-4858-9757-664be2ddce35-istio-podinfo\") pod \"be0a90a3-58fa-4858-9757-664be2ddce35\" (UID: \"be0a90a3-58fa-4858-9757-664be2ddce35\") "
Sep 18 14:13:36.050844 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:36.050683    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be0a90a3-58fa-4858-9757-664be2ddce35-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "be0a90a3-58fa-4858-9757-664be2ddce35" (UID: "be0a90a3-58fa-4858-9757-664be2ddce35"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:13:36.050844 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:36.050702    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be0a90a3-58fa-4858-9757-664be2ddce35-istio-data" (OuterVolumeSpecName: "istio-data") pod "be0a90a3-58fa-4858-9757-664be2ddce35" (UID: "be0a90a3-58fa-4858-9757-664be2ddce35"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:13:36.050844 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:36.050828    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fb515b9-6a12-4753-91d0-77bb90af103e-maas-api-tls\") pod \"maas-api-e2e-derive-a721b5af-77c55bcbf6-th7wp\" (UID: \"8fb515b9-6a12-4753-91d0-77bb90af103e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a721b5af-77c55bcbf6-th7wp"
Sep 18 14:13:36.051010 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:36.050957    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be0a90a3-58fa-4858-9757-664be2ddce35-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:13:36.051010 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:36.050966    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be0a90a3-58fa-4858-9757-664be2ddce35-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "be0a90a3-58fa-4858-9757-664be2ddce35" (UID: "be0a90a3-58fa-4858-9757-664be2ddce35"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:13:36.051010 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:36.050974    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be0a90a3-58fa-4858-9757-664be2ddce35-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:13:36.051167 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:36.051136    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be0a90a3-58fa-4858-9757-664be2ddce35-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "be0a90a3-58fa-4858-9757-664be2ddce35" (UID: "be0a90a3-58fa-4858-9757-664be2ddce35"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:13:36.051250 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:36.051213    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be0a90a3-58fa-4858-9757-664be2ddce35-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "be0a90a3-58fa-4858-9757-664be2ddce35" (UID: "be0a90a3-58fa-4858-9757-664be2ddce35"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:13:36.053036 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:36.053005    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be0a90a3-58fa-4858-9757-664be2ddce35-istio-token" (OuterVolumeSpecName: "istio-token") pod "be0a90a3-58fa-4858-9757-664be2ddce35" (UID: "be0a90a3-58fa-4858-9757-664be2ddce35"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:13:36.053158 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:36.053049    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be0a90a3-58fa-4858-9757-664be2ddce35-kube-api-access-zmcgp" (OuterVolumeSpecName: "kube-api-access-zmcgp") pod "be0a90a3-58fa-4858-9757-664be2ddce35" (UID: "be0a90a3-58fa-4858-9757-664be2ddce35"). InnerVolumeSpecName "kube-api-access-zmcgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:13:36.053433 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:36.053409    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be0a90a3-58fa-4858-9757-664be2ddce35-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "be0a90a3-58fa-4858-9757-664be2ddce35" (UID: "be0a90a3-58fa-4858-9757-664be2ddce35"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:13:36.053524 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:36.053474    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be0a90a3-58fa-4858-9757-664be2ddce35-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "be0a90a3-58fa-4858-9757-664be2ddce35" (UID: "be0a90a3-58fa-4858-9757-664be2ddce35"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 14:13:36.053711 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:36.053693    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fb515b9-6a12-4753-91d0-77bb90af103e-maas-api-tls\") pod \"maas-api-e2e-derive-a721b5af-77c55bcbf6-th7wp\" (UID: \"8fb515b9-6a12-4753-91d0-77bb90af103e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a721b5af-77c55bcbf6-th7wp"
Sep 18 14:13:36.118614 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:36.118513    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:36.151780 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:36.151729    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmcgp\" (UniqueName: \"kubernetes.io/projected/be0a90a3-58fa-4858-9757-664be2ddce35-kube-api-access-zmcgp\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:13:36.155427 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:36.155405    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be0a90a3-58fa-4858-9757-664be2ddce35-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:13:36.155573 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:36.155560    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be0a90a3-58fa-4858-9757-664be2ddce35-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:13:36.155670 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:36.155650    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be0a90a3-58fa-4858-9757-664be2ddce35-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:13:36.155867 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:36.155672    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be0a90a3-58fa-4858-9757-664be2ddce35-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:13:36.155867 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:36.155696    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be0a90a3-58fa-4858-9757-664be2ddce35-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:13:36.155867 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:36.155709    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be0a90a3-58fa-4858-9757-664be2ddce35-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:13:36.305027 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:36.304995    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a721b5af-77c55bcbf6-th7wp"
Sep 18 14:13:36.579049 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:36.579019    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a721b5af-77c55bcbf6-th7wp"]
Sep 18 14:13:36.580989 ip-10-0-143-120 kubenswrapper[2579]: W0918 14:13:36.580955    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fb515b9_6a12_4753_91d0_77bb90af103e.slice/crio-1f2a3dbc5c7f6dfcb08c86dea3e3b2e574f6850dc705e516e15899653516f42e WatchSource:0}: Error finding container 1f2a3dbc5c7f6dfcb08c86dea3e3b2e574f6850dc705e516e15899653516f42e: Status 404 returned error can't find the container with id 1f2a3dbc5c7f6dfcb08c86dea3e3b2e574f6850dc705e516e15899653516f42e
Sep 18 14:13:36.738930 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:36.738899    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a721b5af-77c55bcbf6-th7wp" event={"ID":"8fb515b9-6a12-4753-91d0-77bb90af103e","Type":"ContainerStarted","Data":"1f2a3dbc5c7f6dfcb08c86dea3e3b2e574f6850dc705e516e15899653516f42e"}
Sep 18 14:13:36.740620 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:36.740574    2579 generic.go:356] "Generic (PLEG): container finished" podID="be0a90a3-58fa-4858-9757-664be2ddce35" containerID="a53ca2e6e9eaddcdd0e5c7ca452d4b7bf5dcb28599152535d66c69d86a0511d0" exitCode=0
Sep 18 14:13:36.740736 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:36.740611    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr" event={"ID":"be0a90a3-58fa-4858-9757-664be2ddce35","Type":"ContainerDied","Data":"a53ca2e6e9eaddcdd0e5c7ca452d4b7bf5dcb28599152535d66c69d86a0511d0"}
Sep 18 14:13:36.740736 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:36.740628    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr"
Sep 18 14:13:36.740736 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:36.740673    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr" event={"ID":"be0a90a3-58fa-4858-9757-664be2ddce35","Type":"ContainerDied","Data":"f30209feb530660e1354cef4e5d5196afdae867f175ad161dae64c43eda4f12a"}
Sep 18 14:13:36.740736 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:36.740706    2579 scope.go:117] "RemoveContainer" containerID="a53ca2e6e9eaddcdd0e5c7ca452d4b7bf5dcb28599152535d66c69d86a0511d0"
Sep 18 14:13:36.749969 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:36.749950    2579 scope.go:117] "RemoveContainer" containerID="4e9aaa2bf1fda619b10c720d4d8e33091d41b59daeb3b696084a8bb7a96c1305"
Sep 18 14:13:36.759729 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:36.759713    2579 scope.go:117] "RemoveContainer" containerID="a53ca2e6e9eaddcdd0e5c7ca452d4b7bf5dcb28599152535d66c69d86a0511d0"
Sep 18 14:13:36.760027 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:13:36.760007    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a53ca2e6e9eaddcdd0e5c7ca452d4b7bf5dcb28599152535d66c69d86a0511d0\": container with ID starting with a53ca2e6e9eaddcdd0e5c7ca452d4b7bf5dcb28599152535d66c69d86a0511d0 not found: ID does not exist" containerID="a53ca2e6e9eaddcdd0e5c7ca452d4b7bf5dcb28599152535d66c69d86a0511d0"
Sep 18 14:13:36.760117 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:36.760037    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a53ca2e6e9eaddcdd0e5c7ca452d4b7bf5dcb28599152535d66c69d86a0511d0"} err="failed to get container status \"a53ca2e6e9eaddcdd0e5c7ca452d4b7bf5dcb28599152535d66c69d86a0511d0\": rpc error: code = NotFound desc = could not find container \"a53ca2e6e9eaddcdd0e5c7ca452d4b7bf5dcb28599152535d66c69d86a0511d0\": container with ID starting with a53ca2e6e9eaddcdd0e5c7ca452d4b7bf5dcb28599152535d66c69d86a0511d0 not found: ID does not exist"
Sep 18 14:13:36.760117 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:36.760057    2579 scope.go:117] "RemoveContainer" containerID="4e9aaa2bf1fda619b10c720d4d8e33091d41b59daeb3b696084a8bb7a96c1305"
Sep 18 14:13:36.760351 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:13:36.760332    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e9aaa2bf1fda619b10c720d4d8e33091d41b59daeb3b696084a8bb7a96c1305\": container with ID starting with 4e9aaa2bf1fda619b10c720d4d8e33091d41b59daeb3b696084a8bb7a96c1305 not found: ID does not exist" containerID="4e9aaa2bf1fda619b10c720d4d8e33091d41b59daeb3b696084a8bb7a96c1305"
Sep 18 14:13:36.760391 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:36.760358    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e9aaa2bf1fda619b10c720d4d8e33091d41b59daeb3b696084a8bb7a96c1305"} err="failed to get container status \"4e9aaa2bf1fda619b10c720d4d8e33091d41b59daeb3b696084a8bb7a96c1305\": rpc error: code = NotFound desc = could not find container \"4e9aaa2bf1fda619b10c720d4d8e33091d41b59daeb3b696084a8bb7a96c1305\": container with ID starting with 4e9aaa2bf1fda619b10c720d4d8e33091d41b59daeb3b696084a8bb7a96c1305 not found: ID does not exist"
Sep 18 14:13:36.765856 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:36.765837    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr"]
Sep 18 14:13:36.771878 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:36.771856    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-35b94378-openshift-default-64dfbc5c4-66hqr"]
Sep 18 14:13:37.746996 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:37.746955    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a721b5af-77c55bcbf6-th7wp" event={"ID":"8fb515b9-6a12-4753-91d0-77bb90af103e","Type":"ContainerStarted","Data":"329c834e632c0b4dc485465725d87c2e1bdb87bb3b92b7c97da32455867dead2"}
Sep 18 14:13:37.747478 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:37.747018    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a721b5af-77c55bcbf6-th7wp"
Sep 18 14:13:37.772357 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:37.772288    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a721b5af-77c55bcbf6-th7wp" podStartSLOduration=2.414676771 podStartE2EDuration="2.772262143s" totalImagesPullingTime="357.585372ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:13:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:13:37.772144224 +0000 UTC m=+889.660860155" watchObservedRunningTime="2026-09-18 14:13:37.772262143 +0000 UTC m=+889.660978075"
Sep 18 14:13:38.035784 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:38.035702    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be0a90a3-58fa-4858-9757-664be2ddce35" path="/var/lib/kubelet/pods/be0a90a3-58fa-4858-9757-664be2ddce35/volumes"
Sep 18 14:13:40.155263 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:40.155198    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-ae7021/autores-model-ae7021-kserve-7c4c8c4ff5-l77hk" podUID="1b2fa854-3ea0-482b-b421-dc06a95803dc" containerName="main" containerID="cri-o://5405f411fd64387fe11476b50f4001060504f998f24296592858bb74cd6b4149" gracePeriod=45
Sep 18 14:13:40.415594 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:40.415532    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-ae7021/autores-model-ae7021-kserve-7c4c8c4ff5-l77hk"
Sep 18 14:13:40.488197 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:40.488149    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b2fa854-3ea0-482b-b421-dc06a95803dc-tmp-dir\") pod \"1b2fa854-3ea0-482b-b421-dc06a95803dc\" (UID: \"1b2fa854-3ea0-482b-b421-dc06a95803dc\") "
Sep 18 14:13:40.488379 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:40.488237    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b2fa854-3ea0-482b-b421-dc06a95803dc-tls-certs\") pod \"1b2fa854-3ea0-482b-b421-dc06a95803dc\" (UID: \"1b2fa854-3ea0-482b-b421-dc06a95803dc\") "
Sep 18 14:13:40.488379 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:40.488286    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b2fa854-3ea0-482b-b421-dc06a95803dc-model-cache\") pod \"1b2fa854-3ea0-482b-b421-dc06a95803dc\" (UID: \"1b2fa854-3ea0-482b-b421-dc06a95803dc\") "
Sep 18 14:13:40.488379 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:40.488316    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b2fa854-3ea0-482b-b421-dc06a95803dc-dshm\") pod \"1b2fa854-3ea0-482b-b421-dc06a95803dc\" (UID: \"1b2fa854-3ea0-482b-b421-dc06a95803dc\") "
Sep 18 14:13:40.488379 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:40.488350    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b2fa854-3ea0-482b-b421-dc06a95803dc-home\") pod \"1b2fa854-3ea0-482b-b421-dc06a95803dc\" (UID: \"1b2fa854-3ea0-482b-b421-dc06a95803dc\") "
Sep 18 14:13:40.488612 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:40.488399    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jcvjh\" (UniqueName: \"kubernetes.io/projected/1b2fa854-3ea0-482b-b421-dc06a95803dc-kube-api-access-jcvjh\") pod \"1b2fa854-3ea0-482b-b421-dc06a95803dc\" (UID: \"1b2fa854-3ea0-482b-b421-dc06a95803dc\") "
Sep 18 14:13:40.488612 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:40.488400    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b2fa854-3ea0-482b-b421-dc06a95803dc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1b2fa854-3ea0-482b-b421-dc06a95803dc" (UID: "1b2fa854-3ea0-482b-b421-dc06a95803dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:13:40.488612 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:40.488551    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b2fa854-3ea0-482b-b421-dc06a95803dc-model-cache" (OuterVolumeSpecName: "model-cache") pod "1b2fa854-3ea0-482b-b421-dc06a95803dc" (UID: "1b2fa854-3ea0-482b-b421-dc06a95803dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:13:40.488743 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:40.488644    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b2fa854-3ea0-482b-b421-dc06a95803dc-home" (OuterVolumeSpecName: "home") pod "1b2fa854-3ea0-482b-b421-dc06a95803dc" (UID: "1b2fa854-3ea0-482b-b421-dc06a95803dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:13:40.488807 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:40.488785    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b2fa854-3ea0-482b-b421-dc06a95803dc-tmp-dir\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:13:40.488860 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:40.488807    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b2fa854-3ea0-482b-b421-dc06a95803dc-model-cache\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:13:40.488860 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:40.488816    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b2fa854-3ea0-482b-b421-dc06a95803dc-home\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:13:40.490581 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:40.490551    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b2fa854-3ea0-482b-b421-dc06a95803dc-dshm" (OuterVolumeSpecName: "dshm") pod "1b2fa854-3ea0-482b-b421-dc06a95803dc" (UID: "1b2fa854-3ea0-482b-b421-dc06a95803dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:13:40.490696 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:40.490620    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b2fa854-3ea0-482b-b421-dc06a95803dc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1b2fa854-3ea0-482b-b421-dc06a95803dc" (UID: "1b2fa854-3ea0-482b-b421-dc06a95803dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:13:40.490801 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:40.490783    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b2fa854-3ea0-482b-b421-dc06a95803dc-kube-api-access-jcvjh" (OuterVolumeSpecName: "kube-api-access-jcvjh") pod "1b2fa854-3ea0-482b-b421-dc06a95803dc" (UID: "1b2fa854-3ea0-482b-b421-dc06a95803dc"). InnerVolumeSpecName "kube-api-access-jcvjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:13:40.589379 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:40.589343    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b2fa854-3ea0-482b-b421-dc06a95803dc-tls-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:13:40.589379 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:40.589375    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b2fa854-3ea0-482b-b421-dc06a95803dc-dshm\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:13:40.589379 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:40.589384    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcvjh\" (UniqueName: \"kubernetes.io/projected/1b2fa854-3ea0-482b-b421-dc06a95803dc-kube-api-access-jcvjh\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:13:40.765664 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:40.765580    2579 generic.go:356] "Generic (PLEG): container finished" podID="1b2fa854-3ea0-482b-b421-dc06a95803dc" containerID="5405f411fd64387fe11476b50f4001060504f998f24296592858bb74cd6b4149" exitCode=0
Sep 18 14:13:40.765664 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:40.765626    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-ae7021/autores-model-ae7021-kserve-7c4c8c4ff5-l77hk"
Sep 18 14:13:40.765853 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:40.765669    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-ae7021/autores-model-ae7021-kserve-7c4c8c4ff5-l77hk" event={"ID":"1b2fa854-3ea0-482b-b421-dc06a95803dc","Type":"ContainerDied","Data":"5405f411fd64387fe11476b50f4001060504f998f24296592858bb74cd6b4149"}
Sep 18 14:13:40.765853 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:40.765716    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-ae7021/autores-model-ae7021-kserve-7c4c8c4ff5-l77hk" event={"ID":"1b2fa854-3ea0-482b-b421-dc06a95803dc","Type":"ContainerDied","Data":"92863dd1a26dfefe6405cc03a86946dead4ba22a836ab45b93aa43652844bf4b"}
Sep 18 14:13:40.765853 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:40.765737    2579 scope.go:117] "RemoveContainer" containerID="5405f411fd64387fe11476b50f4001060504f998f24296592858bb74cd6b4149"
Sep 18 14:13:40.774842 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:40.774823    2579 scope.go:117] "RemoveContainer" containerID="5405f411fd64387fe11476b50f4001060504f998f24296592858bb74cd6b4149"
Sep 18 14:13:40.775096 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:13:40.775077    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5405f411fd64387fe11476b50f4001060504f998f24296592858bb74cd6b4149\": container with ID starting with 5405f411fd64387fe11476b50f4001060504f998f24296592858bb74cd6b4149 not found: ID does not exist" containerID="5405f411fd64387fe11476b50f4001060504f998f24296592858bb74cd6b4149"
Sep 18 14:13:40.775142 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:40.775104    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5405f411fd64387fe11476b50f4001060504f998f24296592858bb74cd6b4149"} err="failed to get container status \"5405f411fd64387fe11476b50f4001060504f998f24296592858bb74cd6b4149\": rpc error: code = NotFound desc = could not find container \"5405f411fd64387fe11476b50f4001060504f998f24296592858bb74cd6b4149\": container with ID starting with 5405f411fd64387fe11476b50f4001060504f998f24296592858bb74cd6b4149 not found: ID does not exist"
Sep 18 14:13:40.785597 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:40.785573    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-ae7021/autores-model-ae7021-kserve-7c4c8c4ff5-l77hk"]
Sep 18 14:13:40.789633 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:40.789615    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-ae7021/autores-model-ae7021-kserve-7c4c8c4ff5-l77hk"]
Sep 18 14:13:42.035726 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:42.035694    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b2fa854-3ea0-482b-b421-dc06a95803dc" path="/var/lib/kubelet/pods/1b2fa854-3ea0-482b-b421-dc06a95803dc/volumes"
Sep 18 14:13:43.759813 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:43.759782    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a721b5af-77c55bcbf6-th7wp"
Sep 18 14:13:46.125848 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:46.125819    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:49.983401 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:49.983376    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sg4tv_4626a667-4cd6-468c-8ef3-6e3a925c2608/ovn-acl-logging/0.log"
Sep 18 14:13:49.983864 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:49.983696    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sg4tv_4626a667-4cd6-468c-8ef3-6e3a925c2608/ovn-acl-logging/0.log"
Sep 18 14:13:57.927921 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:57.927891    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:13:59.051124 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:13:59.051091    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:00.007376 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:00.007344    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:00.979310 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:00.979281    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:01.977510 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:01.977484    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:03.165885 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:03.165852    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:03.716294 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:03.716263    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:04.127357 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:04.127320    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:04.256638 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:04.256606    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:04.454007 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:04.453938    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:04.730518 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:04.730449    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825"]
Sep 18 14:14:04.730795 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:04.730782    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be0a90a3-58fa-4858-9757-664be2ddce35" containerName="istio-proxy"
Sep 18 14:14:04.730846 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:04.730796    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="be0a90a3-58fa-4858-9757-664be2ddce35" containerName="istio-proxy"
Sep 18 14:14:04.730846 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:04.730812    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be0a90a3-58fa-4858-9757-664be2ddce35" containerName="istio-proxy"
Sep 18 14:14:04.730846 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:04.730817    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="be0a90a3-58fa-4858-9757-664be2ddce35" containerName="istio-proxy"
Sep 18 14:14:04.730846 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:04.730832    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b2fa854-3ea0-482b-b421-dc06a95803dc" containerName="main"
Sep 18 14:14:04.730846 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:04.730838    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b2fa854-3ea0-482b-b421-dc06a95803dc" containerName="main"
Sep 18 14:14:04.731002 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:04.730905    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b2fa854-3ea0-482b-b421-dc06a95803dc" containerName="main"
Sep 18 14:14:04.731002 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:04.730913    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="be0a90a3-58fa-4858-9757-664be2ddce35" containerName="istio-proxy"
Sep 18 14:14:04.731002 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:04.730918    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="be0a90a3-58fa-4858-9757-664be2ddce35" containerName="istio-proxy"
Sep 18 14:14:04.735477 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:04.735460    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825"
Sep 18 14:14:04.738144 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:04.738124    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-e11b4c-openshift-default-dockercfg-frq7j\""
Sep 18 14:14:04.749928 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:04.749908    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825"]
Sep 18 14:14:04.758469 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:04.758442    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:04.837916 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:04.837891    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:04.884587 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:04.884563    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/458aaf7d-9a1b-4357-beb1-29f71290124d-istio-envoy\") pod \"e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825\" (UID: \"458aaf7d-9a1b-4357-beb1-29f71290124d\") " pod="openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825"
Sep 18 14:14:04.884718 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:04.884597    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/458aaf7d-9a1b-4357-beb1-29f71290124d-istio-data\") pod \"e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825\" (UID: \"458aaf7d-9a1b-4357-beb1-29f71290124d\") " pod="openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825"
Sep 18 14:14:04.884718 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:04.884619    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/458aaf7d-9a1b-4357-beb1-29f71290124d-istio-podinfo\") pod \"e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825\" (UID: \"458aaf7d-9a1b-4357-beb1-29f71290124d\") " pod="openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825"
Sep 18 14:14:04.884718 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:04.884633    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/458aaf7d-9a1b-4357-beb1-29f71290124d-istiod-ca-cert\") pod \"e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825\" (UID: \"458aaf7d-9a1b-4357-beb1-29f71290124d\") " pod="openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825"
Sep 18 14:14:04.884718 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:04.884691    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/458aaf7d-9a1b-4357-beb1-29f71290124d-workload-socket\") pod \"e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825\" (UID: \"458aaf7d-9a1b-4357-beb1-29f71290124d\") " pod="openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825"
Sep 18 14:14:04.884921 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:04.884747    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/458aaf7d-9a1b-4357-beb1-29f71290124d-credential-socket\") pod \"e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825\" (UID: \"458aaf7d-9a1b-4357-beb1-29f71290124d\") " pod="openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825"
Sep 18 14:14:04.884921 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:04.884770    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/458aaf7d-9a1b-4357-beb1-29f71290124d-workload-certs\") pod \"e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825\" (UID: \"458aaf7d-9a1b-4357-beb1-29f71290124d\") " pod="openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825"
Sep 18 14:14:04.884921 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:04.884861    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fbnn\" (UniqueName: \"kubernetes.io/projected/458aaf7d-9a1b-4357-beb1-29f71290124d-kube-api-access-9fbnn\") pod \"e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825\" (UID: \"458aaf7d-9a1b-4357-beb1-29f71290124d\") " pod="openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825"
Sep 18 14:14:04.884921 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:04.884906    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/458aaf7d-9a1b-4357-beb1-29f71290124d-istio-token\") pod \"e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825\" (UID: \"458aaf7d-9a1b-4357-beb1-29f71290124d\") " pod="openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825"
Sep 18 14:14:04.942520 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:04.942500    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:04.985615 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:04.985537    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/458aaf7d-9a1b-4357-beb1-29f71290124d-istio-podinfo\") pod \"e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825\" (UID: \"458aaf7d-9a1b-4357-beb1-29f71290124d\") " pod="openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825"
Sep 18 14:14:04.985615 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:04.985569    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/458aaf7d-9a1b-4357-beb1-29f71290124d-istiod-ca-cert\") pod \"e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825\" (UID: \"458aaf7d-9a1b-4357-beb1-29f71290124d\") " pod="openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825"
Sep 18 14:14:04.985615 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:04.985598    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/458aaf7d-9a1b-4357-beb1-29f71290124d-workload-socket\") pod \"e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825\" (UID: \"458aaf7d-9a1b-4357-beb1-29f71290124d\") " pod="openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825"
Sep 18 14:14:04.985930 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:04.985628    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/458aaf7d-9a1b-4357-beb1-29f71290124d-credential-socket\") pod \"e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825\" (UID: \"458aaf7d-9a1b-4357-beb1-29f71290124d\") " pod="openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825"
Sep 18 14:14:04.985930 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:04.985660    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/458aaf7d-9a1b-4357-beb1-29f71290124d-workload-certs\") pod \"e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825\" (UID: \"458aaf7d-9a1b-4357-beb1-29f71290124d\") " pod="openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825"
Sep 18 14:14:04.985930 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:04.985725    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fbnn\" (UniqueName: \"kubernetes.io/projected/458aaf7d-9a1b-4357-beb1-29f71290124d-kube-api-access-9fbnn\") pod \"e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825\" (UID: \"458aaf7d-9a1b-4357-beb1-29f71290124d\") " pod="openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825"
Sep 18 14:14:04.985930 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:04.985757    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/458aaf7d-9a1b-4357-beb1-29f71290124d-istio-token\") pod \"e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825\" (UID: \"458aaf7d-9a1b-4357-beb1-29f71290124d\") " pod="openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825"
Sep 18 14:14:04.985930 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:04.985832    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/458aaf7d-9a1b-4357-beb1-29f71290124d-istio-envoy\") pod \"e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825\" (UID: \"458aaf7d-9a1b-4357-beb1-29f71290124d\") " pod="openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825"
Sep 18 14:14:04.985930 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:04.985868    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/458aaf7d-9a1b-4357-beb1-29f71290124d-istio-data\") pod \"e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825\" (UID: \"458aaf7d-9a1b-4357-beb1-29f71290124d\") " pod="openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825"
Sep 18 14:14:04.986268 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:04.986051    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/458aaf7d-9a1b-4357-beb1-29f71290124d-workload-socket\") pod \"e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825\" (UID: \"458aaf7d-9a1b-4357-beb1-29f71290124d\") " pod="openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825"
Sep 18 14:14:04.986268 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:04.986100    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/458aaf7d-9a1b-4357-beb1-29f71290124d-credential-socket\") pod \"e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825\" (UID: \"458aaf7d-9a1b-4357-beb1-29f71290124d\") " pod="openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825"
Sep 18 14:14:04.986384 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:04.986334    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/458aaf7d-9a1b-4357-beb1-29f71290124d-workload-certs\") pod \"e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825\" (UID: \"458aaf7d-9a1b-4357-beb1-29f71290124d\") " pod="openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825"
Sep 18 14:14:04.986463 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:04.986443    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/458aaf7d-9a1b-4357-beb1-29f71290124d-istiod-ca-cert\") pod \"e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825\" (UID: \"458aaf7d-9a1b-4357-beb1-29f71290124d\") " pod="openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825"
Sep 18 14:14:04.986586 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:04.986568    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/458aaf7d-9a1b-4357-beb1-29f71290124d-istio-data\") pod \"e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825\" (UID: \"458aaf7d-9a1b-4357-beb1-29f71290124d\") " pod="openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825"
Sep 18 14:14:04.988164 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:04.988144    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/458aaf7d-9a1b-4357-beb1-29f71290124d-istio-podinfo\") pod \"e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825\" (UID: \"458aaf7d-9a1b-4357-beb1-29f71290124d\") " pod="openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825"
Sep 18 14:14:04.988346 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:04.988325    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/458aaf7d-9a1b-4357-beb1-29f71290124d-istio-envoy\") pod \"e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825\" (UID: \"458aaf7d-9a1b-4357-beb1-29f71290124d\") " pod="openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825"
Sep 18 14:14:04.994340 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:04.994317    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/458aaf7d-9a1b-4357-beb1-29f71290124d-istio-token\") pod \"e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825\" (UID: \"458aaf7d-9a1b-4357-beb1-29f71290124d\") " pod="openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825"
Sep 18 14:14:04.994660 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:04.994638    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fbnn\" (UniqueName: \"kubernetes.io/projected/458aaf7d-9a1b-4357-beb1-29f71290124d-kube-api-access-9fbnn\") pod \"e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825\" (UID: \"458aaf7d-9a1b-4357-beb1-29f71290124d\") " pod="openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825"
Sep 18 14:14:05.044911 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:05.044888    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825"
Sep 18 14:14:05.324675 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:05.324651    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825"]
Sep 18 14:14:05.327072 ip-10-0-143-120 kubenswrapper[2579]: W0918 14:14:05.327046    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod458aaf7d_9a1b_4357_beb1_29f71290124d.slice/crio-f25b9d5ffa71918f0bf14f73f665091202861f4b537e88e6a5572cda31bf7666 WatchSource:0}: Error finding container f25b9d5ffa71918f0bf14f73f665091202861f4b537e88e6a5572cda31bf7666: Status 404 returned error can't find the container with id f25b9d5ffa71918f0bf14f73f665091202861f4b537e88e6a5572cda31bf7666
Sep 18 14:14:05.329413 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:05.329370    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:14:05.329512 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:05.329451    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:14:05.329512 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:05.329497    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:14:05.841509 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:05.841473    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:05.885765 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:05.885732    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825" event={"ID":"458aaf7d-9a1b-4357-beb1-29f71290124d","Type":"ContainerStarted","Data":"c478a4446ec4cedd29ad66d9170132404318ceb55b0341a7fe79888e90993ff4"}
Sep 18 14:14:05.885899 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:05.885788    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825" event={"ID":"458aaf7d-9a1b-4357-beb1-29f71290124d","Type":"ContainerStarted","Data":"f25b9d5ffa71918f0bf14f73f665091202861f4b537e88e6a5572cda31bf7666"}
Sep 18 14:14:05.914322 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:05.914274    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825" podStartSLOduration=1.9142592189999998 podStartE2EDuration="1.914259219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:14:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:14:05.911241292 +0000 UTC m=+917.799957222" watchObservedRunningTime="2026-09-18 14:14:05.914259219 +0000 UTC m=+917.802975150"
Sep 18 14:14:06.045631 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:06.045602    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825"
Sep 18 14:14:06.050548 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:06.050525    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825"
Sep 18 14:14:06.748406 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:06.748370    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:06.891067 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:06.891037    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825"
Sep 18 14:14:06.891886 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:06.891866    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825"
Sep 18 14:14:07.542394 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:07.542349    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:07.994342 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:07.994309    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:08.080461 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:08.080426    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:08.580189 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:08.577387    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:09.610904 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:09.610866    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:10.522232 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:10.522186    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:11.460691 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:11.460655    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:12.220338 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:12.220306    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:13.257067 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:13.257027    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:13.334633 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:13.334598    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:13.496432 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:13.496398    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:13.784996 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:13.784971    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:13.882945 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:13.882914    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:14.135476 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:14.135447    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a721b5af-77c55bcbf6-th7wp"]
Sep 18 14:14:14.135646 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:14.135624    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a721b5af-77c55bcbf6-th7wp" podUID="8fb515b9-6a12-4753-91d0-77bb90af103e" containerName="maas-api" containerID="cri-o://329c834e632c0b4dc485465725d87c2e1bdb87bb3b92b7c97da32455867dead2" gracePeriod=30
Sep 18 14:14:14.157542 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:14.157512    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:14.255861 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:14.255833    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:14.331614 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:14.331573    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:14.386490 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:14.386442    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a721b5af-77c55bcbf6-th7wp"
Sep 18 14:14:14.416577 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:14.416536    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:14.567631 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:14.567599    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fb515b9-6a12-4753-91d0-77bb90af103e-openshift-service-ca\") pod \"8fb515b9-6a12-4753-91d0-77bb90af103e\" (UID: \"8fb515b9-6a12-4753-91d0-77bb90af103e\") "
Sep 18 14:14:14.567821 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:14.567689    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fb515b9-6a12-4753-91d0-77bb90af103e-metrics-certs\") pod \"8fb515b9-6a12-4753-91d0-77bb90af103e\" (UID: \"8fb515b9-6a12-4753-91d0-77bb90af103e\") "
Sep 18 14:14:14.567821 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:14.567811    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fb515b9-6a12-4753-91d0-77bb90af103e-maas-api-tls\") pod \"8fb515b9-6a12-4753-91d0-77bb90af103e\" (UID: \"8fb515b9-6a12-4753-91d0-77bb90af103e\") "
Sep 18 14:14:14.567941 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:14.567835    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gfjgw\" (UniqueName: \"kubernetes.io/projected/8fb515b9-6a12-4753-91d0-77bb90af103e-kube-api-access-gfjgw\") pod \"8fb515b9-6a12-4753-91d0-77bb90af103e\" (UID: \"8fb515b9-6a12-4753-91d0-77bb90af103e\") "
Sep 18 14:14:14.567941 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:14.567926    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fb515b9-6a12-4753-91d0-77bb90af103e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8fb515b9-6a12-4753-91d0-77bb90af103e" (UID: "8fb515b9-6a12-4753-91d0-77bb90af103e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:14:14.568098 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:14.568079    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fb515b9-6a12-4753-91d0-77bb90af103e-openshift-service-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:14:14.569806 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:14.569771    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fb515b9-6a12-4753-91d0-77bb90af103e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8fb515b9-6a12-4753-91d0-77bb90af103e" (UID: "8fb515b9-6a12-4753-91d0-77bb90af103e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:14:14.569924 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:14.569879    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fb515b9-6a12-4753-91d0-77bb90af103e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8fb515b9-6a12-4753-91d0-77bb90af103e" (UID: "8fb515b9-6a12-4753-91d0-77bb90af103e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:14:14.569924 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:14.569890    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fb515b9-6a12-4753-91d0-77bb90af103e-kube-api-access-gfjgw" (OuterVolumeSpecName: "kube-api-access-gfjgw") pod "8fb515b9-6a12-4753-91d0-77bb90af103e" (UID: "8fb515b9-6a12-4753-91d0-77bb90af103e"). InnerVolumeSpecName "kube-api-access-gfjgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:14:14.578811 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:14.578779    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:14.665145 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:14.665063    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:14.669302 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:14.669284    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fb515b9-6a12-4753-91d0-77bb90af103e-maas-api-tls\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:14:14.669374 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:14.669304    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfjgw\" (UniqueName: \"kubernetes.io/projected/8fb515b9-6a12-4753-91d0-77bb90af103e-kube-api-access-gfjgw\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:14:14.669374 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:14.669313    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fb515b9-6a12-4753-91d0-77bb90af103e-metrics-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:14:14.927469 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:14.927388    2579 generic.go:356] "Generic (PLEG): container finished" podID="8fb515b9-6a12-4753-91d0-77bb90af103e" containerID="329c834e632c0b4dc485465725d87c2e1bdb87bb3b92b7c97da32455867dead2" exitCode=0
Sep 18 14:14:14.927469 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:14.927440    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a721b5af-77c55bcbf6-th7wp"
Sep 18 14:14:14.927664 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:14.927476    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a721b5af-77c55bcbf6-th7wp" event={"ID":"8fb515b9-6a12-4753-91d0-77bb90af103e","Type":"ContainerDied","Data":"329c834e632c0b4dc485465725d87c2e1bdb87bb3b92b7c97da32455867dead2"}
Sep 18 14:14:14.927664 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:14.927553    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a721b5af-77c55bcbf6-th7wp" event={"ID":"8fb515b9-6a12-4753-91d0-77bb90af103e","Type":"ContainerDied","Data":"1f2a3dbc5c7f6dfcb08c86dea3e3b2e574f6850dc705e516e15899653516f42e"}
Sep 18 14:14:14.927664 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:14.927578    2579 scope.go:117] "RemoveContainer" containerID="329c834e632c0b4dc485465725d87c2e1bdb87bb3b92b7c97da32455867dead2"
Sep 18 14:14:14.936123 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:14.936107    2579 scope.go:117] "RemoveContainer" containerID="329c834e632c0b4dc485465725d87c2e1bdb87bb3b92b7c97da32455867dead2"
Sep 18 14:14:14.936393 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:14:14.936375    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"329c834e632c0b4dc485465725d87c2e1bdb87bb3b92b7c97da32455867dead2\": container with ID starting with 329c834e632c0b4dc485465725d87c2e1bdb87bb3b92b7c97da32455867dead2 not found: ID does not exist" containerID="329c834e632c0b4dc485465725d87c2e1bdb87bb3b92b7c97da32455867dead2"
Sep 18 14:14:14.936448 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:14.936401    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"329c834e632c0b4dc485465725d87c2e1bdb87bb3b92b7c97da32455867dead2"} err="failed to get container status \"329c834e632c0b4dc485465725d87c2e1bdb87bb3b92b7c97da32455867dead2\": rpc error: code = NotFound desc = could not find container \"329c834e632c0b4dc485465725d87c2e1bdb87bb3b92b7c97da32455867dead2\": container with ID starting with 329c834e632c0b4dc485465725d87c2e1bdb87bb3b92b7c97da32455867dead2 not found: ID does not exist"
Sep 18 14:14:14.951655 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:14.951636    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a721b5af-77c55bcbf6-th7wp"]
Sep 18 14:14:14.963290 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:14.963270    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a721b5af-77c55bcbf6-th7wp"]
Sep 18 14:14:15.292480 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:15.292407    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:15.389725 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:15.389701    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:15.547626 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:15.547597    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:16.034755 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:16.034666    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fb515b9-6a12-4753-91d0-77bb90af103e" path="/var/lib/kubelet/pods/8fb515b9-6a12-4753-91d0-77bb90af103e/volumes"
Sep 18 14:14:16.541333 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:16.541298    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:16.843203 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:16.843145    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:17.359813 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:17.359787    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:19.316359 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:19.316329    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:19.733728 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:19.733656    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:23.821885 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:23.821850    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:23.932949 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:23.932920    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:24.033930 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:24.033898    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:24.139665 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:24.139581    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:24.518234 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:24.518127    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:33.022433 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:33.022361    2579 generic.go:356] "Generic (PLEG): container finished" podID="458aaf7d-9a1b-4357-beb1-29f71290124d" containerID="c478a4446ec4cedd29ad66d9170132404318ceb55b0341a7fe79888e90993ff4" exitCode=0
Sep 18 14:14:33.022433 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:33.022419    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825" event={"ID":"458aaf7d-9a1b-4357-beb1-29f71290124d","Type":"ContainerDied","Data":"c478a4446ec4cedd29ad66d9170132404318ceb55b0341a7fe79888e90993ff4"}
Sep 18 14:14:33.022802 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:33.022751    2579 scope.go:117] "RemoveContainer" containerID="c478a4446ec4cedd29ad66d9170132404318ceb55b0341a7fe79888e90993ff4"
Sep 18 14:14:33.023791 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:33.023763    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:14:33.023846 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:33.023833    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:14:33.023882 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:33.023868    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:14:33.410112 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:33.410042    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:34.033956 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:34.033919    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825" event={"ID":"458aaf7d-9a1b-4357-beb1-29f71290124d","Type":"ContainerStarted","Data":"2ef0abf2d10151796630988a8589c46f0820d32762b6edc2faebbec827f25421"}
Sep 18 14:14:34.046015 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:34.045994    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825"
Sep 18 14:14:34.047100 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:34.047082    2579 patch_prober.go:28] interesting pod/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Sep 18 14:14:34.047150 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:34.047124    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825" podUID="458aaf7d-9a1b-4357-beb1-29f71290124d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Sep 18 14:14:35.045353 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:35.045321    2579 patch_prober.go:28] interesting pod/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Sep 18 14:14:35.045732 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:35.045380    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825" podUID="458aaf7d-9a1b-4357-beb1-29f71290124d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Sep 18 14:14:35.829527 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:35.829493    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:36.045853 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:36.045821    2579 patch_prober.go:28] interesting pod/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Sep 18 14:14:36.046266 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:36.045871    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825" podUID="458aaf7d-9a1b-4357-beb1-29f71290124d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Sep 18 14:14:36.552698 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:36.552664    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:36.891292 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:36.891212    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825"
Sep 18 14:14:37.045971 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:37.045936    2579 patch_prober.go:28] interesting pod/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Sep 18 14:14:37.046443 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:37.045990    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825" podUID="458aaf7d-9a1b-4357-beb1-29f71290124d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Sep 18 14:14:37.316910 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:37.316883    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:38.009465 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:38.009431    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:38.045925 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:38.045893    2579 patch_prober.go:28] interesting pod/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Sep 18 14:14:38.046107 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:38.045951    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825" podUID="458aaf7d-9a1b-4357-beb1-29f71290124d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Sep 18 14:14:38.404097 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:38.404070    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:39.078730 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:39.078693    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825"
Sep 18 14:14:39.080189 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:39.080150    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825"
Sep 18 14:14:39.187725 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:39.187688    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:40.185937 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:40.185906    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:45.878815 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:45.878785    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:50.452703 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:50.452620    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:56.617915 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:56.617864    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:56.764283 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:56.764254    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:56.862820 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:56.862785    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:59.803932 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:59.803897    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-ee8a08/test-model-ee8a08-kserve-d868b947f-8l6w6"]
Sep 18 14:14:59.804503 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:59.804485    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fb515b9-6a12-4753-91d0-77bb90af103e" containerName="maas-api"
Sep 18 14:14:59.804589 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:59.804506    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fb515b9-6a12-4753-91d0-77bb90af103e" containerName="maas-api"
Sep 18 14:14:59.804644 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:59.804605    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fb515b9-6a12-4753-91d0-77bb90af103e" containerName="maas-api"
Sep 18 14:14:59.808674 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:59.808655    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ee8a08/test-model-ee8a08-kserve-d868b947f-8l6w6"
Sep 18 14:14:59.812146 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:59.812120    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ee8a08\"/\"test-model-ee8a08-kserve-self-signed-certs\""
Sep 18 14:14:59.812279 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:59.812199    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ee8a08\"/\"default-dockercfg-74rtk\""
Sep 18 14:14:59.812279 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:59.812252    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ee8a08\"/\"openshift-service-ca.crt\""
Sep 18 14:14:59.812472 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:59.812458    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ee8a08\"/\"kube-root-ca.crt\""
Sep 18 14:14:59.826260 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:59.826239    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ee8a08/test-model-ee8a08-kserve-d868b947f-8l6w6"]
Sep 18 14:14:59.917716 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:59.917689    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:14:59.957867 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:59.957842    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19670d14-3e75-4660-9c15-3854636aa005-tmp-dir\") pod \"test-model-ee8a08-kserve-d868b947f-8l6w6\" (UID: \"19670d14-3e75-4660-9c15-3854636aa005\") " pod="ai-tenant-e2e-inf-b-ee8a08/test-model-ee8a08-kserve-d868b947f-8l6w6"
Sep 18 14:14:59.958006 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:59.957888    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19670d14-3e75-4660-9c15-3854636aa005-home\") pod \"test-model-ee8a08-kserve-d868b947f-8l6w6\" (UID: \"19670d14-3e75-4660-9c15-3854636aa005\") " pod="ai-tenant-e2e-inf-b-ee8a08/test-model-ee8a08-kserve-d868b947f-8l6w6"
Sep 18 14:14:59.958006 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:59.957908    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19670d14-3e75-4660-9c15-3854636aa005-dshm\") pod \"test-model-ee8a08-kserve-d868b947f-8l6w6\" (UID: \"19670d14-3e75-4660-9c15-3854636aa005\") " pod="ai-tenant-e2e-inf-b-ee8a08/test-model-ee8a08-kserve-d868b947f-8l6w6"
Sep 18 14:14:59.958006 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:59.957931    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19670d14-3e75-4660-9c15-3854636aa005-model-cache\") pod \"test-model-ee8a08-kserve-d868b947f-8l6w6\" (UID: \"19670d14-3e75-4660-9c15-3854636aa005\") " pod="ai-tenant-e2e-inf-b-ee8a08/test-model-ee8a08-kserve-d868b947f-8l6w6"
Sep 18 14:14:59.958006 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:59.957985    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqbgc\" (UniqueName: \"kubernetes.io/projected/19670d14-3e75-4660-9c15-3854636aa005-kube-api-access-mqbgc\") pod \"test-model-ee8a08-kserve-d868b947f-8l6w6\" (UID: \"19670d14-3e75-4660-9c15-3854636aa005\") " pod="ai-tenant-e2e-inf-b-ee8a08/test-model-ee8a08-kserve-d868b947f-8l6w6"
Sep 18 14:14:59.958166 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:14:59.958079    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19670d14-3e75-4660-9c15-3854636aa005-tls-certs\") pod \"test-model-ee8a08-kserve-d868b947f-8l6w6\" (UID: \"19670d14-3e75-4660-9c15-3854636aa005\") " pod="ai-tenant-e2e-inf-b-ee8a08/test-model-ee8a08-kserve-d868b947f-8l6w6"
Sep 18 14:15:00.058812 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:00.058751    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19670d14-3e75-4660-9c15-3854636aa005-tls-certs\") pod \"test-model-ee8a08-kserve-d868b947f-8l6w6\" (UID: \"19670d14-3e75-4660-9c15-3854636aa005\") " pod="ai-tenant-e2e-inf-b-ee8a08/test-model-ee8a08-kserve-d868b947f-8l6w6"
Sep 18 14:15:00.058812 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:00.058788    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19670d14-3e75-4660-9c15-3854636aa005-tmp-dir\") pod \"test-model-ee8a08-kserve-d868b947f-8l6w6\" (UID: \"19670d14-3e75-4660-9c15-3854636aa005\") " pod="ai-tenant-e2e-inf-b-ee8a08/test-model-ee8a08-kserve-d868b947f-8l6w6"
Sep 18 14:15:00.058968 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:00.058819    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19670d14-3e75-4660-9c15-3854636aa005-home\") pod \"test-model-ee8a08-kserve-d868b947f-8l6w6\" (UID: \"19670d14-3e75-4660-9c15-3854636aa005\") " pod="ai-tenant-e2e-inf-b-ee8a08/test-model-ee8a08-kserve-d868b947f-8l6w6"
Sep 18 14:15:00.059114 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:00.059092    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19670d14-3e75-4660-9c15-3854636aa005-dshm\") pod \"test-model-ee8a08-kserve-d868b947f-8l6w6\" (UID: \"19670d14-3e75-4660-9c15-3854636aa005\") " pod="ai-tenant-e2e-inf-b-ee8a08/test-model-ee8a08-kserve-d868b947f-8l6w6"
Sep 18 14:15:00.059439 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:00.059130    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19670d14-3e75-4660-9c15-3854636aa005-model-cache\") pod \"test-model-ee8a08-kserve-d868b947f-8l6w6\" (UID: \"19670d14-3e75-4660-9c15-3854636aa005\") " pod="ai-tenant-e2e-inf-b-ee8a08/test-model-ee8a08-kserve-d868b947f-8l6w6"
Sep 18 14:15:00.059439 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:00.059182    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqbgc\" (UniqueName: \"kubernetes.io/projected/19670d14-3e75-4660-9c15-3854636aa005-kube-api-access-mqbgc\") pod \"test-model-ee8a08-kserve-d868b947f-8l6w6\" (UID: \"19670d14-3e75-4660-9c15-3854636aa005\") " pod="ai-tenant-e2e-inf-b-ee8a08/test-model-ee8a08-kserve-d868b947f-8l6w6"
Sep 18 14:15:00.059439 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:00.059213    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19670d14-3e75-4660-9c15-3854636aa005-home\") pod \"test-model-ee8a08-kserve-d868b947f-8l6w6\" (UID: \"19670d14-3e75-4660-9c15-3854636aa005\") " pod="ai-tenant-e2e-inf-b-ee8a08/test-model-ee8a08-kserve-d868b947f-8l6w6"
Sep 18 14:15:00.059439 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:00.059243    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19670d14-3e75-4660-9c15-3854636aa005-tmp-dir\") pod \"test-model-ee8a08-kserve-d868b947f-8l6w6\" (UID: \"19670d14-3e75-4660-9c15-3854636aa005\") " pod="ai-tenant-e2e-inf-b-ee8a08/test-model-ee8a08-kserve-d868b947f-8l6w6"
Sep 18 14:15:00.059674 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:00.059528    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19670d14-3e75-4660-9c15-3854636aa005-model-cache\") pod \"test-model-ee8a08-kserve-d868b947f-8l6w6\" (UID: \"19670d14-3e75-4660-9c15-3854636aa005\") " pod="ai-tenant-e2e-inf-b-ee8a08/test-model-ee8a08-kserve-d868b947f-8l6w6"
Sep 18 14:15:00.061263 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:00.061240    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19670d14-3e75-4660-9c15-3854636aa005-dshm\") pod \"test-model-ee8a08-kserve-d868b947f-8l6w6\" (UID: \"19670d14-3e75-4660-9c15-3854636aa005\") " pod="ai-tenant-e2e-inf-b-ee8a08/test-model-ee8a08-kserve-d868b947f-8l6w6"
Sep 18 14:15:00.061514 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:00.061499    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19670d14-3e75-4660-9c15-3854636aa005-tls-certs\") pod \"test-model-ee8a08-kserve-d868b947f-8l6w6\" (UID: \"19670d14-3e75-4660-9c15-3854636aa005\") " pod="ai-tenant-e2e-inf-b-ee8a08/test-model-ee8a08-kserve-d868b947f-8l6w6"
Sep 18 14:15:00.068417 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:00.068399    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqbgc\" (UniqueName: \"kubernetes.io/projected/19670d14-3e75-4660-9c15-3854636aa005-kube-api-access-mqbgc\") pod \"test-model-ee8a08-kserve-d868b947f-8l6w6\" (UID: \"19670d14-3e75-4660-9c15-3854636aa005\") " pod="ai-tenant-e2e-inf-b-ee8a08/test-model-ee8a08-kserve-d868b947f-8l6w6"
Sep 18 14:15:00.117506 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:00.117479    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ee8a08/test-model-ee8a08-kserve-d868b947f-8l6w6"
Sep 18 14:15:00.398813 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:00.398788    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ee8a08/test-model-ee8a08-kserve-d868b947f-8l6w6"]
Sep 18 14:15:00.400815 ip-10-0-143-120 kubenswrapper[2579]: W0918 14:15:00.400788    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19670d14_3e75_4660_9c15_3854636aa005.slice/crio-450c29ec37687d2358b863fb34557c00481f37a40536e1fd977e8b879f05eddd WatchSource:0}: Error finding container 450c29ec37687d2358b863fb34557c00481f37a40536e1fd977e8b879f05eddd: Status 404 returned error can't find the container with id 450c29ec37687d2358b863fb34557c00481f37a40536e1fd977e8b879f05eddd
Sep 18 14:15:01.157379 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:01.157343    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ee8a08/test-model-ee8a08-kserve-d868b947f-8l6w6" event={"ID":"19670d14-3e75-4660-9c15-3854636aa005","Type":"ContainerStarted","Data":"f51f4fe9133f7d9e6baf8fefc341616012d1b0b12f8ab984db0b0d1d55a5e388"}
Sep 18 14:15:01.157854 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:01.157391    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ee8a08/test-model-ee8a08-kserve-d868b947f-8l6w6" event={"ID":"19670d14-3e75-4660-9c15-3854636aa005","Type":"ContainerStarted","Data":"450c29ec37687d2358b863fb34557c00481f37a40536e1fd977e8b879f05eddd"}
Sep 18 14:15:01.200272 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:01.200226    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-ee8a08/test-model-ee8a08-kserve-d868b947f-8l6w6" podStartSLOduration=2.200209975 podStartE2EDuration="2.200209975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:14:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:15:01.198657221 +0000 UTC m=+973.087373220" watchObservedRunningTime="2026-09-18 14:15:01.200209975 +0000 UTC m=+973.088925905"
Sep 18 14:15:10.118128 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:10.118089    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-ee8a08/test-model-ee8a08-kserve-d868b947f-8l6w6"
Sep 18 14:15:10.118736 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:10.118154    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ee8a08/test-model-ee8a08-kserve-d868b947f-8l6w6"
Sep 18 14:15:10.130883 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:10.130861    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-ee8a08/test-model-ee8a08-kserve-d868b947f-8l6w6"
Sep 18 14:15:10.213650 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:10.213622    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ee8a08/test-model-ee8a08-kserve-d868b947f-8l6w6"
Sep 18 14:15:12.724770 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:12.724723    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:15:18.712553 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:18.712518    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:15:18.821104 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:18.821073    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:15:18.914468 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:18.914433    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:15:26.713774 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:26.713738    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:15:29.420480 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:29.420450    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:15:30.198595 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:30.198565    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:15:36.411703 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:36.411653    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:15:36.646391 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:36.646356    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:15:36.860384 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:36.860300    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:15:37.386183 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:37.386144    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:15:37.490647 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:37.490610    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:15:39.005620 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:39.005590    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:15:45.061864 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:45.061821    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:15:45.165106 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:45.165072    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:15:45.800161 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:45.800125    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:15:50.875591 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:50.875555    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:15:52.292368 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:52.292336    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:15:53.253637 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:53.253608    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:15:53.351678 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:53.351649    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:15:53.480511 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:53.480478    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:15:53.767127 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:53.767086    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2"]
Sep 18 14:15:53.770499 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:53.770476    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2"
Sep 18 14:15:53.772619 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:53.772599    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-834b7e2a-openshift-default-dockercfg-d5rrp\""
Sep 18 14:15:53.794778 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:53.794756    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2"]
Sep 18 14:15:53.830334 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:53.830313    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:15:53.915967 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:53.915931    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-istio-envoy\") pod \"e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2\" (UID: \"f7609088-ae6a-40ee-8c95-bcd5e27fa0ce\") " pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2"
Sep 18 14:15:53.915967 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:53.915968    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-istio-token\") pod \"e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2\" (UID: \"f7609088-ae6a-40ee-8c95-bcd5e27fa0ce\") " pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2"
Sep 18 14:15:53.916236 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:53.915991    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-workload-socket\") pod \"e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2\" (UID: \"f7609088-ae6a-40ee-8c95-bcd5e27fa0ce\") " pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2"
Sep 18 14:15:53.916236 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:53.916077    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-istiod-ca-cert\") pod \"e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2\" (UID: \"f7609088-ae6a-40ee-8c95-bcd5e27fa0ce\") " pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2"
Sep 18 14:15:53.916236 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:53.916104    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6c5q\" (UniqueName: \"kubernetes.io/projected/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-kube-api-access-s6c5q\") pod \"e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2\" (UID: \"f7609088-ae6a-40ee-8c95-bcd5e27fa0ce\") " pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2"
Sep 18 14:15:53.916236 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:53.916148    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-workload-certs\") pod \"e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2\" (UID: \"f7609088-ae6a-40ee-8c95-bcd5e27fa0ce\") " pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2"
Sep 18 14:15:53.916236 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:53.916216    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-istio-podinfo\") pod \"e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2\" (UID: \"f7609088-ae6a-40ee-8c95-bcd5e27fa0ce\") " pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2"
Sep 18 14:15:53.916476 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:53.916265    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-credential-socket\") pod \"e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2\" (UID: \"f7609088-ae6a-40ee-8c95-bcd5e27fa0ce\") " pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2"
Sep 18 14:15:53.916476 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:53.916331    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-istio-data\") pod \"e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2\" (UID: \"f7609088-ae6a-40ee-8c95-bcd5e27fa0ce\") " pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2"
Sep 18 14:15:53.957212 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:53.957187    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:15:54.017334 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:54.017269    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-istio-data\") pod \"e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2\" (UID: \"f7609088-ae6a-40ee-8c95-bcd5e27fa0ce\") " pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2"
Sep 18 14:15:54.017334 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:54.017315    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-istio-envoy\") pod \"e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2\" (UID: \"f7609088-ae6a-40ee-8c95-bcd5e27fa0ce\") " pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2"
Sep 18 14:15:54.017529 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:54.017335    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-istio-token\") pod \"e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2\" (UID: \"f7609088-ae6a-40ee-8c95-bcd5e27fa0ce\") " pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2"
Sep 18 14:15:54.017529 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:54.017356    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-workload-socket\") pod \"e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2\" (UID: \"f7609088-ae6a-40ee-8c95-bcd5e27fa0ce\") " pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2"
Sep 18 14:15:54.017529 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:54.017384    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-istiod-ca-cert\") pod \"e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2\" (UID: \"f7609088-ae6a-40ee-8c95-bcd5e27fa0ce\") " pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2"
Sep 18 14:15:54.017529 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:54.017408    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6c5q\" (UniqueName: \"kubernetes.io/projected/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-kube-api-access-s6c5q\") pod \"e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2\" (UID: \"f7609088-ae6a-40ee-8c95-bcd5e27fa0ce\") " pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2"
Sep 18 14:15:54.017769 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:54.017567    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-workload-certs\") pod \"e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2\" (UID: \"f7609088-ae6a-40ee-8c95-bcd5e27fa0ce\") " pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2"
Sep 18 14:15:54.017769 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:54.017603    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-istio-podinfo\") pod \"e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2\" (UID: \"f7609088-ae6a-40ee-8c95-bcd5e27fa0ce\") " pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2"
Sep 18 14:15:54.017769 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:54.017644    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-credential-socket\") pod \"e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2\" (UID: \"f7609088-ae6a-40ee-8c95-bcd5e27fa0ce\") " pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2"
Sep 18 14:15:54.017769 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:54.017720    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-istio-data\") pod \"e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2\" (UID: \"f7609088-ae6a-40ee-8c95-bcd5e27fa0ce\") " pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2"
Sep 18 14:15:54.017945 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:54.017816    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-workload-socket\") pod \"e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2\" (UID: \"f7609088-ae6a-40ee-8c95-bcd5e27fa0ce\") " pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2"
Sep 18 14:15:54.017945 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:54.017918    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-workload-certs\") pod \"e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2\" (UID: \"f7609088-ae6a-40ee-8c95-bcd5e27fa0ce\") " pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2"
Sep 18 14:15:54.018025 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:54.018002    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-credential-socket\") pod \"e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2\" (UID: \"f7609088-ae6a-40ee-8c95-bcd5e27fa0ce\") " pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2"
Sep 18 14:15:54.018183 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:54.018147    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-istiod-ca-cert\") pod \"e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2\" (UID: \"f7609088-ae6a-40ee-8c95-bcd5e27fa0ce\") " pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2"
Sep 18 14:15:54.019560 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:54.019537    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-istio-envoy\") pod \"e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2\" (UID: \"f7609088-ae6a-40ee-8c95-bcd5e27fa0ce\") " pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2"
Sep 18 14:15:54.019973 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:54.019957    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-istio-podinfo\") pod \"e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2\" (UID: \"f7609088-ae6a-40ee-8c95-bcd5e27fa0ce\") " pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2"
Sep 18 14:15:54.025608 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:54.025587    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6c5q\" (UniqueName: \"kubernetes.io/projected/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-kube-api-access-s6c5q\") pod \"e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2\" (UID: \"f7609088-ae6a-40ee-8c95-bcd5e27fa0ce\") " pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2"
Sep 18 14:15:54.025982 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:54.025962    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-istio-token\") pod \"e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2\" (UID: \"f7609088-ae6a-40ee-8c95-bcd5e27fa0ce\") " pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2"
Sep 18 14:15:54.081025 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:54.081004    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2"
Sep 18 14:15:54.477804 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:54.477775    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2"]
Sep 18 14:15:54.481361 ip-10-0-143-120 kubenswrapper[2579]: W0918 14:15:54.481330    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7609088_ae6a_40ee_8c95_bcd5e27fa0ce.slice/crio-a0a1d65004e820b184fe2f8ffbc2b4502bf6008f54571dd523e06ff6635e6157 WatchSource:0}: Error finding container a0a1d65004e820b184fe2f8ffbc2b4502bf6008f54571dd523e06ff6635e6157: Status 404 returned error can't find the container with id a0a1d65004e820b184fe2f8ffbc2b4502bf6008f54571dd523e06ff6635e6157
Sep 18 14:15:54.483593 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:54.483559    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:15:54.483683 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:54.483623    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:15:54.483683 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:54.483651    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:15:54.682515 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:54.682443    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:15:55.414080 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:55.414046    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2" event={"ID":"f7609088-ae6a-40ee-8c95-bcd5e27fa0ce","Type":"ContainerStarted","Data":"4d97e0d8ac4508f9002e5ce74ca516cd96434ed15050b31f16ba9d37843521c2"}
Sep 18 14:15:55.414287 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:55.414098    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2" event={"ID":"f7609088-ae6a-40ee-8c95-bcd5e27fa0ce","Type":"ContainerStarted","Data":"a0a1d65004e820b184fe2f8ffbc2b4502bf6008f54571dd523e06ff6635e6157"}
Sep 18 14:15:55.458738 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:55.458691    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2" podStartSLOduration=2.458672278 podStartE2EDuration="2.458672278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:15:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:15:55.455426435 +0000 UTC m=+1027.344142368" watchObservedRunningTime="2026-09-18 14:15:55.458672278 +0000 UTC m=+1027.347388211"
Sep 18 14:15:55.949342 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:55.949314    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:15:56.081204 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:56.081117    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2"
Sep 18 14:15:56.082586 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:56.082553    2579 patch_prober.go:28] interesting pod/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 18 14:15:56.082720 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:56.082615    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2" podUID="f7609088-ae6a-40ee-8c95-bcd5e27fa0ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 18 14:15:56.756504 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:56.756459    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:15:57.085765 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:57.085739    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2"
Sep 18 14:15:57.423044 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:57.422947    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2"
Sep 18 14:15:57.423757 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:57.423740    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2"
Sep 18 14:15:58.095606 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:15:58.095577    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:00.045823 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:00.045786    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:01.581781 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:01.581741    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:01.713667 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:01.713636    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:01.925220 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:01.925138    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:03.037877 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:03.037845    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:03.838119 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:03.838080    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:04.639955 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:04.639920    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:05.437282 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:05.437236    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:06.211103 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:06.211065    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-834b7e2a-5857d9d49c-24gdv"]
Sep 18 14:16:06.214131 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:06.214109    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-834b7e2a-5857d9d49c-24gdv"
Sep 18 14:16:06.218269 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:06.218244    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-834b7e2a-serving-cert\""
Sep 18 14:16:06.228180 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:06.228148    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-834b7e2a-5857d9d49c-24gdv"]
Sep 18 14:16:06.255489 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:06.255460    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:06.329505 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:06.329480    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aca703a8-e0fa-49c7-8b56-3eb922ff38f5-maas-api-tls\") pod \"maas-api-e2e-mt-834b7e2a-5857d9d49c-24gdv\" (UID: \"aca703a8-e0fa-49c7-8b56-3eb922ff38f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-834b7e2a-5857d9d49c-24gdv"
Sep 18 14:16:06.329664 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:06.329539    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aca703a8-e0fa-49c7-8b56-3eb922ff38f5-openshift-service-ca\") pod \"maas-api-e2e-mt-834b7e2a-5857d9d49c-24gdv\" (UID: \"aca703a8-e0fa-49c7-8b56-3eb922ff38f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-834b7e2a-5857d9d49c-24gdv"
Sep 18 14:16:06.329664 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:06.329636    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aca703a8-e0fa-49c7-8b56-3eb922ff38f5-metrics-certs\") pod \"maas-api-e2e-mt-834b7e2a-5857d9d49c-24gdv\" (UID: \"aca703a8-e0fa-49c7-8b56-3eb922ff38f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-834b7e2a-5857d9d49c-24gdv"
Sep 18 14:16:06.329798 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:06.329707    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn8xh\" (UniqueName: \"kubernetes.io/projected/aca703a8-e0fa-49c7-8b56-3eb922ff38f5-kube-api-access-vn8xh\") pod \"maas-api-e2e-mt-834b7e2a-5857d9d49c-24gdv\" (UID: \"aca703a8-e0fa-49c7-8b56-3eb922ff38f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-834b7e2a-5857d9d49c-24gdv"
Sep 18 14:16:06.356522 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:06.356483    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:06.431031 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:06.430997    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn8xh\" (UniqueName: \"kubernetes.io/projected/aca703a8-e0fa-49c7-8b56-3eb922ff38f5-kube-api-access-vn8xh\") pod \"maas-api-e2e-mt-834b7e2a-5857d9d49c-24gdv\" (UID: \"aca703a8-e0fa-49c7-8b56-3eb922ff38f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-834b7e2a-5857d9d49c-24gdv"
Sep 18 14:16:06.431209 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:06.431108    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aca703a8-e0fa-49c7-8b56-3eb922ff38f5-maas-api-tls\") pod \"maas-api-e2e-mt-834b7e2a-5857d9d49c-24gdv\" (UID: \"aca703a8-e0fa-49c7-8b56-3eb922ff38f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-834b7e2a-5857d9d49c-24gdv"
Sep 18 14:16:06.431209 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:06.431159    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aca703a8-e0fa-49c7-8b56-3eb922ff38f5-openshift-service-ca\") pod \"maas-api-e2e-mt-834b7e2a-5857d9d49c-24gdv\" (UID: \"aca703a8-e0fa-49c7-8b56-3eb922ff38f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-834b7e2a-5857d9d49c-24gdv"
Sep 18 14:16:06.431330 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:06.431227    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aca703a8-e0fa-49c7-8b56-3eb922ff38f5-metrics-certs\") pod \"maas-api-e2e-mt-834b7e2a-5857d9d49c-24gdv\" (UID: \"aca703a8-e0fa-49c7-8b56-3eb922ff38f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-834b7e2a-5857d9d49c-24gdv"
Sep 18 14:16:06.431330 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:16:06.431264    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-834b7e2a-serving-cert: secret "maas-api-e2e-mt-834b7e2a-serving-cert" not found
Sep 18 14:16:06.431451 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:16:06.431355    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aca703a8-e0fa-49c7-8b56-3eb922ff38f5-maas-api-tls podName:aca703a8-e0fa-49c7-8b56-3eb922ff38f5 nodeName:}" failed. No retries permitted until 2026-09-18 14:16:06.931327902 +0000 UTC m=+1038.820043834 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/aca703a8-e0fa-49c7-8b56-3eb922ff38f5-maas-api-tls") pod "maas-api-e2e-mt-834b7e2a-5857d9d49c-24gdv" (UID: "aca703a8-e0fa-49c7-8b56-3eb922ff38f5") : secret "maas-api-e2e-mt-834b7e2a-serving-cert" not found
Sep 18 14:16:06.431807 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:06.431788    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aca703a8-e0fa-49c7-8b56-3eb922ff38f5-openshift-service-ca\") pod \"maas-api-e2e-mt-834b7e2a-5857d9d49c-24gdv\" (UID: \"aca703a8-e0fa-49c7-8b56-3eb922ff38f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-834b7e2a-5857d9d49c-24gdv"
Sep 18 14:16:06.433665 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:06.433642    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aca703a8-e0fa-49c7-8b56-3eb922ff38f5-metrics-certs\") pod \"maas-api-e2e-mt-834b7e2a-5857d9d49c-24gdv\" (UID: \"aca703a8-e0fa-49c7-8b56-3eb922ff38f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-834b7e2a-5857d9d49c-24gdv"
Sep 18 14:16:06.442691 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:06.442667    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn8xh\" (UniqueName: \"kubernetes.io/projected/aca703a8-e0fa-49c7-8b56-3eb922ff38f5-kube-api-access-vn8xh\") pod \"maas-api-e2e-mt-834b7e2a-5857d9d49c-24gdv\" (UID: \"aca703a8-e0fa-49c7-8b56-3eb922ff38f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-834b7e2a-5857d9d49c-24gdv"
Sep 18 14:16:06.520527 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:06.520469    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:06.816688 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:06.816659    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:06.914941 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:06.914912    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:06.934877 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:06.934852    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aca703a8-e0fa-49c7-8b56-3eb922ff38f5-maas-api-tls\") pod \"maas-api-e2e-mt-834b7e2a-5857d9d49c-24gdv\" (UID: \"aca703a8-e0fa-49c7-8b56-3eb922ff38f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-834b7e2a-5857d9d49c-24gdv"
Sep 18 14:16:06.937208 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:06.937186    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aca703a8-e0fa-49c7-8b56-3eb922ff38f5-maas-api-tls\") pod \"maas-api-e2e-mt-834b7e2a-5857d9d49c-24gdv\" (UID: \"aca703a8-e0fa-49c7-8b56-3eb922ff38f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-834b7e2a-5857d9d49c-24gdv"
Sep 18 14:16:07.124906 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:07.124811    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-834b7e2a-5857d9d49c-24gdv"
Sep 18 14:16:07.176207 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:07.176153    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:07.269583 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:07.269545    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:07.345468 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:07.345438    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:07.414927 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:07.414899    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-834b7e2a-5857d9d49c-24gdv"]
Sep 18 14:16:07.416811 ip-10-0-143-120 kubenswrapper[2579]: W0918 14:16:07.416778    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaca703a8_e0fa_49c7_8b56_3eb922ff38f5.slice/crio-3a1ccc7737a7e83efd9e881df6960aafd1ac25104b6c0530b94fcf94d1591f68 WatchSource:0}: Error finding container 3a1ccc7737a7e83efd9e881df6960aafd1ac25104b6c0530b94fcf94d1591f68: Status 404 returned error can't find the container with id 3a1ccc7737a7e83efd9e881df6960aafd1ac25104b6c0530b94fcf94d1591f68
Sep 18 14:16:07.418918 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:07.418891    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 14:16:07.448930 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:07.448902    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:07.474220 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:07.474189    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-834b7e2a-5857d9d49c-24gdv" event={"ID":"aca703a8-e0fa-49c7-8b56-3eb922ff38f5","Type":"ContainerStarted","Data":"3a1ccc7737a7e83efd9e881df6960aafd1ac25104b6c0530b94fcf94d1591f68"}
Sep 18 14:16:07.593454 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:07.593422    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:07.783859 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:07.783786    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:07.904718 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:07.904692    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:08.481874 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:08.481839    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-834b7e2a-5857d9d49c-24gdv" event={"ID":"aca703a8-e0fa-49c7-8b56-3eb922ff38f5","Type":"ContainerStarted","Data":"3f5a266d6f1d12e58cd6767342368990d3bb254da2bad68c05b85483ecb84b99"}
Sep 18 14:16:08.482261 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:08.481902    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-834b7e2a-5857d9d49c-24gdv"
Sep 18 14:16:08.500964 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:08.500898    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-834b7e2a-5857d9d49c-24gdv" podStartSLOduration=2.118098784 podStartE2EDuration="2.500874326s" totalImagesPullingTime="382.775542ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:16:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:16:08.500401148 +0000 UTC m=+1040.389117082" watchObservedRunningTime="2026-09-18 14:16:08.500874326 +0000 UTC m=+1040.389590258"
Sep 18 14:16:08.879339 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:08.879304    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:09.871752 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:09.871719    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:10.686553 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:10.686525    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:11.679682 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:11.679653    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:12.588012 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:12.587969    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:13.834336 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:13.834301    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:14.492700 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:14.492671    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-834b7e2a-5857d9d49c-24gdv"
Sep 18 14:16:15.077116 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:15.077085    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:15.186539 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:15.186503    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:15.327802 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:15.327726    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825"]
Sep 18 14:16:15.327956 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:15.327918    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825" podUID="458aaf7d-9a1b-4357-beb1-29f71290124d" containerName="istio-proxy" containerID="cri-o://2ef0abf2d10151796630988a8589c46f0820d32762b6edc2faebbec827f25421" gracePeriod=30
Sep 18 14:16:15.346060 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:15.346022    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:15.798703 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:15.798678    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:16.159402 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:16.159315    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:16.290676 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:16.290644    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:16.740866 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:16.740832    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:16.843379 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:16.843350    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:17.643856 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:17.643828    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:17.736560 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:17.736528    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:17.858835 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:17.858806    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:20.541311 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:20.541281    2579 generic.go:356] "Generic (PLEG): container finished" podID="458aaf7d-9a1b-4357-beb1-29f71290124d" containerID="2ef0abf2d10151796630988a8589c46f0820d32762b6edc2faebbec827f25421" exitCode=0
Sep 18 14:16:20.541698 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:20.541355    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825" event={"ID":"458aaf7d-9a1b-4357-beb1-29f71290124d","Type":"ContainerDied","Data":"2ef0abf2d10151796630988a8589c46f0820d32762b6edc2faebbec827f25421"}
Sep 18 14:16:20.541698 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:20.541426    2579 scope.go:117] "RemoveContainer" containerID="c478a4446ec4cedd29ad66d9170132404318ceb55b0341a7fe79888e90993ff4"
Sep 18 14:16:20.594489 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:20.594433    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825"
Sep 18 14:16:20.655505 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:20.655475    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/458aaf7d-9a1b-4357-beb1-29f71290124d-credential-socket\") pod \"458aaf7d-9a1b-4357-beb1-29f71290124d\" (UID: \"458aaf7d-9a1b-4357-beb1-29f71290124d\") "
Sep 18 14:16:20.655667 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:20.655523    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/458aaf7d-9a1b-4357-beb1-29f71290124d-istio-token\") pod \"458aaf7d-9a1b-4357-beb1-29f71290124d\" (UID: \"458aaf7d-9a1b-4357-beb1-29f71290124d\") "
Sep 18 14:16:20.655667 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:20.655554    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/458aaf7d-9a1b-4357-beb1-29f71290124d-istio-envoy\") pod \"458aaf7d-9a1b-4357-beb1-29f71290124d\" (UID: \"458aaf7d-9a1b-4357-beb1-29f71290124d\") "
Sep 18 14:16:20.655667 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:20.655595    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/458aaf7d-9a1b-4357-beb1-29f71290124d-istiod-ca-cert\") pod \"458aaf7d-9a1b-4357-beb1-29f71290124d\" (UID: \"458aaf7d-9a1b-4357-beb1-29f71290124d\") "
Sep 18 14:16:20.655839 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:20.655667    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/458aaf7d-9a1b-4357-beb1-29f71290124d-istio-podinfo\") pod \"458aaf7d-9a1b-4357-beb1-29f71290124d\" (UID: \"458aaf7d-9a1b-4357-beb1-29f71290124d\") "
Sep 18 14:16:20.655839 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:20.655717    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/458aaf7d-9a1b-4357-beb1-29f71290124d-istio-data\") pod \"458aaf7d-9a1b-4357-beb1-29f71290124d\" (UID: \"458aaf7d-9a1b-4357-beb1-29f71290124d\") "
Sep 18 14:16:20.655839 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:20.655768    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9fbnn\" (UniqueName: \"kubernetes.io/projected/458aaf7d-9a1b-4357-beb1-29f71290124d-kube-api-access-9fbnn\") pod \"458aaf7d-9a1b-4357-beb1-29f71290124d\" (UID: \"458aaf7d-9a1b-4357-beb1-29f71290124d\") "
Sep 18 14:16:20.655839 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:20.655797    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/458aaf7d-9a1b-4357-beb1-29f71290124d-workload-certs\") pod \"458aaf7d-9a1b-4357-beb1-29f71290124d\" (UID: \"458aaf7d-9a1b-4357-beb1-29f71290124d\") "
Sep 18 14:16:20.655839 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:20.655824    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/458aaf7d-9a1b-4357-beb1-29f71290124d-workload-socket\") pod \"458aaf7d-9a1b-4357-beb1-29f71290124d\" (UID: \"458aaf7d-9a1b-4357-beb1-29f71290124d\") "
Sep 18 14:16:20.656461 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:20.656315    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/458aaf7d-9a1b-4357-beb1-29f71290124d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "458aaf7d-9a1b-4357-beb1-29f71290124d" (UID: "458aaf7d-9a1b-4357-beb1-29f71290124d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:16:20.656461 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:20.656420    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/458aaf7d-9a1b-4357-beb1-29f71290124d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "458aaf7d-9a1b-4357-beb1-29f71290124d" (UID: "458aaf7d-9a1b-4357-beb1-29f71290124d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:16:20.656620 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:20.656485    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/458aaf7d-9a1b-4357-beb1-29f71290124d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "458aaf7d-9a1b-4357-beb1-29f71290124d" (UID: "458aaf7d-9a1b-4357-beb1-29f71290124d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:16:20.658324 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:20.658290    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/458aaf7d-9a1b-4357-beb1-29f71290124d-istio-data" (OuterVolumeSpecName: "istio-data") pod "458aaf7d-9a1b-4357-beb1-29f71290124d" (UID: "458aaf7d-9a1b-4357-beb1-29f71290124d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:16:20.658324 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:20.658307    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/458aaf7d-9a1b-4357-beb1-29f71290124d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "458aaf7d-9a1b-4357-beb1-29f71290124d" (UID: "458aaf7d-9a1b-4357-beb1-29f71290124d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:16:20.659016 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:20.658785    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/458aaf7d-9a1b-4357-beb1-29f71290124d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "458aaf7d-9a1b-4357-beb1-29f71290124d" (UID: "458aaf7d-9a1b-4357-beb1-29f71290124d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:16:20.659016 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:20.658957    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/458aaf7d-9a1b-4357-beb1-29f71290124d-istio-token" (OuterVolumeSpecName: "istio-token") pod "458aaf7d-9a1b-4357-beb1-29f71290124d" (UID: "458aaf7d-9a1b-4357-beb1-29f71290124d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:16:20.667109 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:20.666943    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/458aaf7d-9a1b-4357-beb1-29f71290124d-kube-api-access-9fbnn" (OuterVolumeSpecName: "kube-api-access-9fbnn") pod "458aaf7d-9a1b-4357-beb1-29f71290124d" (UID: "458aaf7d-9a1b-4357-beb1-29f71290124d"). InnerVolumeSpecName "kube-api-access-9fbnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:16:20.667951 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:20.667922    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/458aaf7d-9a1b-4357-beb1-29f71290124d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "458aaf7d-9a1b-4357-beb1-29f71290124d" (UID: "458aaf7d-9a1b-4357-beb1-29f71290124d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 14:16:20.756907 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:20.756885    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/458aaf7d-9a1b-4357-beb1-29f71290124d-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:16:20.756991 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:20.756909    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/458aaf7d-9a1b-4357-beb1-29f71290124d-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:16:20.756991 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:20.756922    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/458aaf7d-9a1b-4357-beb1-29f71290124d-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:16:20.756991 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:20.756929    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/458aaf7d-9a1b-4357-beb1-29f71290124d-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:16:20.756991 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:20.756937    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/458aaf7d-9a1b-4357-beb1-29f71290124d-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:16:20.756991 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:20.756944    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/458aaf7d-9a1b-4357-beb1-29f71290124d-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:16:20.756991 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:20.756952    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fbnn\" (UniqueName: \"kubernetes.io/projected/458aaf7d-9a1b-4357-beb1-29f71290124d-kube-api-access-9fbnn\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:16:20.756991 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:20.756961    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/458aaf7d-9a1b-4357-beb1-29f71290124d-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:16:20.756991 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:20.756969    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/458aaf7d-9a1b-4357-beb1-29f71290124d-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:16:21.547320 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:21.547293    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825"
Sep 18 14:16:21.547750 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:21.547292    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825" event={"ID":"458aaf7d-9a1b-4357-beb1-29f71290124d","Type":"ContainerDied","Data":"f25b9d5ffa71918f0bf14f73f665091202861f4b537e88e6a5572cda31bf7666"}
Sep 18 14:16:21.547750 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:21.547417    2579 scope.go:117] "RemoveContainer" containerID="2ef0abf2d10151796630988a8589c46f0820d32762b6edc2faebbec827f25421"
Sep 18 14:16:21.584970 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:21.584932    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825"]
Sep 18 14:16:21.585844 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:21.585810    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-e11b4c-openshift-default-7d48d9588d-zn825"]
Sep 18 14:16:22.035292 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:22.035217    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="458aaf7d-9a1b-4357-beb1-29f71290124d" path="/var/lib/kubelet/pods/458aaf7d-9a1b-4357-beb1-29f71290124d/volumes"
Sep 18 14:16:27.420186 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:27.420076    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:28.464333 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:28.464296    2579 patch_prober.go:28] interesting pod/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 18 14:16:28.464784 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:28.464359    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2" podUID="f7609088-ae6a-40ee-8c95-bcd5e27fa0ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 18 14:16:28.837442 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:28.837399    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:29.075890 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:29.075854    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:29.595440 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:29.595410    2579 generic.go:356] "Generic (PLEG): container finished" podID="f7609088-ae6a-40ee-8c95-bcd5e27fa0ce" containerID="4d97e0d8ac4508f9002e5ce74ca516cd96434ed15050b31f16ba9d37843521c2" exitCode=0
Sep 18 14:16:29.595806 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:29.595479    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2" event={"ID":"f7609088-ae6a-40ee-8c95-bcd5e27fa0ce","Type":"ContainerDied","Data":"4d97e0d8ac4508f9002e5ce74ca516cd96434ed15050b31f16ba9d37843521c2"}
Sep 18 14:16:29.595870 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:29.595851    2579 scope.go:117] "RemoveContainer" containerID="4d97e0d8ac4508f9002e5ce74ca516cd96434ed15050b31f16ba9d37843521c2"
Sep 18 14:16:29.596961 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:29.596935    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:16:29.597042 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:29.596999    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:16:29.597042 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:29.597033    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:16:29.792235 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:29.792128    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:30.581751 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:30.581672    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:30.602740 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:30.602712    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2" event={"ID":"f7609088-ae6a-40ee-8c95-bcd5e27fa0ce","Type":"ContainerStarted","Data":"0cd18f8a55e944583bcebe0087ccf613ce44c5b34e9bd52b8f5e617912b05e8e"}
Sep 18 14:16:31.082035 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:31.082000    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2"
Sep 18 14:16:31.082035 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:31.082025    2579 patch_prober.go:28] interesting pod/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 18 14:16:31.082328 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:31.082065    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2" podUID="f7609088-ae6a-40ee-8c95-bcd5e27fa0ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 18 14:16:31.547544 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:31.547486    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:31.650452 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:31.650408    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:32.081642 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:32.081606    2579 patch_prober.go:28] interesting pod/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 18 14:16:32.081923 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:32.081889    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2" podUID="f7609088-ae6a-40ee-8c95-bcd5e27fa0ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 18 14:16:32.260334 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:32.260302    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:32.367066 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:32.366982    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:32.683135 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:32.683011    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:32.788626 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:32.788598    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:33.082207 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:33.082067    2579 patch_prober.go:28] interesting pod/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 18 14:16:33.082207 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:33.082122    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2" podUID="f7609088-ae6a-40ee-8c95-bcd5e27fa0ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 18 14:16:33.646544 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:33.646509    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:34.081497 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:34.081461    2579 patch_prober.go:28] interesting pod/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 18 14:16:34.082018 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:34.081538    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2" podUID="f7609088-ae6a-40ee-8c95-bcd5e27fa0ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 18 14:16:35.082157 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:35.082122    2579 patch_prober.go:28] interesting pod/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 18 14:16:35.082662 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:35.082205    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2" podUID="f7609088-ae6a-40ee-8c95-bcd5e27fa0ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 18 14:16:36.085418 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:36.085389    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2"
Sep 18 14:16:36.085878 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:36.085773    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2"
Sep 18 14:16:36.086331 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:36.086313    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2"
Sep 18 14:16:38.617995 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:38.617960    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:38.716330 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:38.716302    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:40.070450 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:40.070417    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:41.128703 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:41.128665    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:42.310080 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:42.310050    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:43.141226 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:43.141150    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:43.982876 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:43.982842    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:44.835303 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:44.835268    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:45.763228 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:45.763165    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:48.041410 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:48.041381    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:49.159844 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:49.159807    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:50.147514 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:50.147478    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:50.940949 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:50.940911    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-834b7e2a-5857d9d49c-24gdv"]
Sep 18 14:16:50.941355 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:50.941110    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-834b7e2a-5857d9d49c-24gdv" podUID="aca703a8-e0fa-49c7-8b56-3eb922ff38f5" containerName="maas-api" containerID="cri-o://3f5a266d6f1d12e58cd6767342368990d3bb254da2bad68c05b85483ecb84b99" gracePeriod=30
Sep 18 14:16:51.011334 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:51.011302    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:51.137400 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:51.137366    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:51.189751 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:51.189725    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-834b7e2a-5857d9d49c-24gdv"
Sep 18 14:16:51.230444 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:51.230382    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aca703a8-e0fa-49c7-8b56-3eb922ff38f5-metrics-certs\") pod \"aca703a8-e0fa-49c7-8b56-3eb922ff38f5\" (UID: \"aca703a8-e0fa-49c7-8b56-3eb922ff38f5\") "
Sep 18 14:16:51.230444 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:51.230427    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vn8xh\" (UniqueName: \"kubernetes.io/projected/aca703a8-e0fa-49c7-8b56-3eb922ff38f5-kube-api-access-vn8xh\") pod \"aca703a8-e0fa-49c7-8b56-3eb922ff38f5\" (UID: \"aca703a8-e0fa-49c7-8b56-3eb922ff38f5\") "
Sep 18 14:16:51.230583 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:51.230386    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:51.230583 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:51.230478    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aca703a8-e0fa-49c7-8b56-3eb922ff38f5-maas-api-tls\") pod \"aca703a8-e0fa-49c7-8b56-3eb922ff38f5\" (UID: \"aca703a8-e0fa-49c7-8b56-3eb922ff38f5\") "
Sep 18 14:16:51.230583 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:51.230551    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aca703a8-e0fa-49c7-8b56-3eb922ff38f5-openshift-service-ca\") pod \"aca703a8-e0fa-49c7-8b56-3eb922ff38f5\" (UID: \"aca703a8-e0fa-49c7-8b56-3eb922ff38f5\") "
Sep 18 14:16:51.230960 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:51.230938    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aca703a8-e0fa-49c7-8b56-3eb922ff38f5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "aca703a8-e0fa-49c7-8b56-3eb922ff38f5" (UID: "aca703a8-e0fa-49c7-8b56-3eb922ff38f5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:16:51.232387 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:51.232362    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aca703a8-e0fa-49c7-8b56-3eb922ff38f5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "aca703a8-e0fa-49c7-8b56-3eb922ff38f5" (UID: "aca703a8-e0fa-49c7-8b56-3eb922ff38f5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:16:51.232546 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:51.232520    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aca703a8-e0fa-49c7-8b56-3eb922ff38f5-kube-api-access-vn8xh" (OuterVolumeSpecName: "kube-api-access-vn8xh") pod "aca703a8-e0fa-49c7-8b56-3eb922ff38f5" (UID: "aca703a8-e0fa-49c7-8b56-3eb922ff38f5"). InnerVolumeSpecName "kube-api-access-vn8xh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:16:51.232636 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:51.232566    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aca703a8-e0fa-49c7-8b56-3eb922ff38f5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "aca703a8-e0fa-49c7-8b56-3eb922ff38f5" (UID: "aca703a8-e0fa-49c7-8b56-3eb922ff38f5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:16:51.331340 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:51.331318    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aca703a8-e0fa-49c7-8b56-3eb922ff38f5-maas-api-tls\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:16:51.331340 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:51.331339    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aca703a8-e0fa-49c7-8b56-3eb922ff38f5-openshift-service-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:16:51.331466 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:51.331349    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aca703a8-e0fa-49c7-8b56-3eb922ff38f5-metrics-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:16:51.331466 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:51.331358    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn8xh\" (UniqueName: \"kubernetes.io/projected/aca703a8-e0fa-49c7-8b56-3eb922ff38f5-kube-api-access-vn8xh\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:16:51.344375 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:51.344348    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:51.497106 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:51.497006    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:51.707223 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:51.707157    2579 generic.go:356] "Generic (PLEG): container finished" podID="aca703a8-e0fa-49c7-8b56-3eb922ff38f5" containerID="3f5a266d6f1d12e58cd6767342368990d3bb254da2bad68c05b85483ecb84b99" exitCode=0
Sep 18 14:16:51.707223 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:51.707215    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-834b7e2a-5857d9d49c-24gdv" event={"ID":"aca703a8-e0fa-49c7-8b56-3eb922ff38f5","Type":"ContainerDied","Data":"3f5a266d6f1d12e58cd6767342368990d3bb254da2bad68c05b85483ecb84b99"}
Sep 18 14:16:51.707443 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:51.707237    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-834b7e2a-5857d9d49c-24gdv"
Sep 18 14:16:51.707443 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:51.707281    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-834b7e2a-5857d9d49c-24gdv" event={"ID":"aca703a8-e0fa-49c7-8b56-3eb922ff38f5","Type":"ContainerDied","Data":"3a1ccc7737a7e83efd9e881df6960aafd1ac25104b6c0530b94fcf94d1591f68"}
Sep 18 14:16:51.707443 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:51.707315    2579 scope.go:117] "RemoveContainer" containerID="3f5a266d6f1d12e58cd6767342368990d3bb254da2bad68c05b85483ecb84b99"
Sep 18 14:16:51.723288 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:51.723260    2579 scope.go:117] "RemoveContainer" containerID="3f5a266d6f1d12e58cd6767342368990d3bb254da2bad68c05b85483ecb84b99"
Sep 18 14:16:51.723911 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:16:51.723881    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f5a266d6f1d12e58cd6767342368990d3bb254da2bad68c05b85483ecb84b99\": container with ID starting with 3f5a266d6f1d12e58cd6767342368990d3bb254da2bad68c05b85483ecb84b99 not found: ID does not exist" containerID="3f5a266d6f1d12e58cd6767342368990d3bb254da2bad68c05b85483ecb84b99"
Sep 18 14:16:51.724023 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:51.723923    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f5a266d6f1d12e58cd6767342368990d3bb254da2bad68c05b85483ecb84b99"} err="failed to get container status \"3f5a266d6f1d12e58cd6767342368990d3bb254da2bad68c05b85483ecb84b99\": rpc error: code = NotFound desc = could not find container \"3f5a266d6f1d12e58cd6767342368990d3bb254da2bad68c05b85483ecb84b99\": container with ID starting with 3f5a266d6f1d12e58cd6767342368990d3bb254da2bad68c05b85483ecb84b99 not found: ID does not exist"
Sep 18 14:16:51.737635 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:51.737603    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-834b7e2a-5857d9d49c-24gdv"]
Sep 18 14:16:51.741500 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:51.741468    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:51.747321 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:51.747251    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-834b7e2a-5857d9d49c-24gdv"]
Sep 18 14:16:51.835984 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:51.835945    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:52.036249 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:52.036142    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aca703a8-e0fa-49c7-8b56-3eb922ff38f5" path="/var/lib/kubelet/pods/aca703a8-e0fa-49c7-8b56-3eb922ff38f5/volumes"
Sep 18 14:16:52.773232 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:52.773202    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:53.905221 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:53.905186    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:54.530560 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:54.530524    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-4d9222/distinct-w0-4d9222-kserve-77444cbdf6-bz5lg"]
Sep 18 14:16:54.530910 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:54.530898    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="458aaf7d-9a1b-4357-beb1-29f71290124d" containerName="istio-proxy"
Sep 18 14:16:54.530957 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:54.530912    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="458aaf7d-9a1b-4357-beb1-29f71290124d" containerName="istio-proxy"
Sep 18 14:16:54.530957 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:54.530941    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aca703a8-e0fa-49c7-8b56-3eb922ff38f5" containerName="maas-api"
Sep 18 14:16:54.530957 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:54.530950    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="aca703a8-e0fa-49c7-8b56-3eb922ff38f5" containerName="maas-api"
Sep 18 14:16:54.531058 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:54.531002    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="458aaf7d-9a1b-4357-beb1-29f71290124d" containerName="istio-proxy"
Sep 18 14:16:54.531058 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:54.531014    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="aca703a8-e0fa-49c7-8b56-3eb922ff38f5" containerName="maas-api"
Sep 18 14:16:54.531058 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:54.531024    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="458aaf7d-9a1b-4357-beb1-29f71290124d" containerName="istio-proxy"
Sep 18 14:16:54.535115 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:54.535096    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4d9222/distinct-w0-4d9222-kserve-77444cbdf6-bz5lg"
Sep 18 14:16:54.538546 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:54.538526    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-4d9222\"/\"default-dockercfg-st5nr\""
Sep 18 14:16:54.538661 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:54.538582    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-4d9222\"/\"distinct-w0-4d9222-kserve-self-signed-certs\""
Sep 18 14:16:54.538661 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:54.538598    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-4d9222\"/\"kube-root-ca.crt\""
Sep 18 14:16:54.538661 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:54.538654    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-4d9222\"/\"openshift-service-ca.crt\""
Sep 18 14:16:54.557101 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:54.557077    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4d9222/distinct-w0-4d9222-kserve-77444cbdf6-bz5lg"]
Sep 18 14:16:54.559262 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:54.559243    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/272988d2-cade-46e9-8fef-ad40e429f9b4-dshm\") pod \"distinct-w0-4d9222-kserve-77444cbdf6-bz5lg\" (UID: \"272988d2-cade-46e9-8fef-ad40e429f9b4\") " pod="e2e-models-e2e-worker-w0-4d9222/distinct-w0-4d9222-kserve-77444cbdf6-bz5lg"
Sep 18 14:16:54.559358 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:54.559278    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/272988d2-cade-46e9-8fef-ad40e429f9b4-home\") pod \"distinct-w0-4d9222-kserve-77444cbdf6-bz5lg\" (UID: \"272988d2-cade-46e9-8fef-ad40e429f9b4\") " pod="e2e-models-e2e-worker-w0-4d9222/distinct-w0-4d9222-kserve-77444cbdf6-bz5lg"
Sep 18 14:16:54.559358 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:54.559324    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfffg\" (UniqueName: \"kubernetes.io/projected/272988d2-cade-46e9-8fef-ad40e429f9b4-kube-api-access-bfffg\") pod \"distinct-w0-4d9222-kserve-77444cbdf6-bz5lg\" (UID: \"272988d2-cade-46e9-8fef-ad40e429f9b4\") " pod="e2e-models-e2e-worker-w0-4d9222/distinct-w0-4d9222-kserve-77444cbdf6-bz5lg"
Sep 18 14:16:54.559433 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:54.559370    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/272988d2-cade-46e9-8fef-ad40e429f9b4-model-cache\") pod \"distinct-w0-4d9222-kserve-77444cbdf6-bz5lg\" (UID: \"272988d2-cade-46e9-8fef-ad40e429f9b4\") " pod="e2e-models-e2e-worker-w0-4d9222/distinct-w0-4d9222-kserve-77444cbdf6-bz5lg"
Sep 18 14:16:54.559488 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:54.559426    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/272988d2-cade-46e9-8fef-ad40e429f9b4-tls-certs\") pod \"distinct-w0-4d9222-kserve-77444cbdf6-bz5lg\" (UID: \"272988d2-cade-46e9-8fef-ad40e429f9b4\") " pod="e2e-models-e2e-worker-w0-4d9222/distinct-w0-4d9222-kserve-77444cbdf6-bz5lg"
Sep 18 14:16:54.559488 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:54.559461    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/272988d2-cade-46e9-8fef-ad40e429f9b4-tmp-dir\") pod \"distinct-w0-4d9222-kserve-77444cbdf6-bz5lg\" (UID: \"272988d2-cade-46e9-8fef-ad40e429f9b4\") " pod="e2e-models-e2e-worker-w0-4d9222/distinct-w0-4d9222-kserve-77444cbdf6-bz5lg"
Sep 18 14:16:54.660573 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:54.660538    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/272988d2-cade-46e9-8fef-ad40e429f9b4-home\") pod \"distinct-w0-4d9222-kserve-77444cbdf6-bz5lg\" (UID: \"272988d2-cade-46e9-8fef-ad40e429f9b4\") " pod="e2e-models-e2e-worker-w0-4d9222/distinct-w0-4d9222-kserve-77444cbdf6-bz5lg"
Sep 18 14:16:54.660728 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:54.660608    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfffg\" (UniqueName: \"kubernetes.io/projected/272988d2-cade-46e9-8fef-ad40e429f9b4-kube-api-access-bfffg\") pod \"distinct-w0-4d9222-kserve-77444cbdf6-bz5lg\" (UID: \"272988d2-cade-46e9-8fef-ad40e429f9b4\") " pod="e2e-models-e2e-worker-w0-4d9222/distinct-w0-4d9222-kserve-77444cbdf6-bz5lg"
Sep 18 14:16:54.660728 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:54.660647    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/272988d2-cade-46e9-8fef-ad40e429f9b4-model-cache\") pod \"distinct-w0-4d9222-kserve-77444cbdf6-bz5lg\" (UID: \"272988d2-cade-46e9-8fef-ad40e429f9b4\") " pod="e2e-models-e2e-worker-w0-4d9222/distinct-w0-4d9222-kserve-77444cbdf6-bz5lg"
Sep 18 14:16:54.660728 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:54.660690    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/272988d2-cade-46e9-8fef-ad40e429f9b4-tls-certs\") pod \"distinct-w0-4d9222-kserve-77444cbdf6-bz5lg\" (UID: \"272988d2-cade-46e9-8fef-ad40e429f9b4\") " pod="e2e-models-e2e-worker-w0-4d9222/distinct-w0-4d9222-kserve-77444cbdf6-bz5lg"
Sep 18 14:16:54.660728 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:54.660719    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/272988d2-cade-46e9-8fef-ad40e429f9b4-tmp-dir\") pod \"distinct-w0-4d9222-kserve-77444cbdf6-bz5lg\" (UID: \"272988d2-cade-46e9-8fef-ad40e429f9b4\") " pod="e2e-models-e2e-worker-w0-4d9222/distinct-w0-4d9222-kserve-77444cbdf6-bz5lg"
Sep 18 14:16:54.660949 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:54.660760    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/272988d2-cade-46e9-8fef-ad40e429f9b4-dshm\") pod \"distinct-w0-4d9222-kserve-77444cbdf6-bz5lg\" (UID: \"272988d2-cade-46e9-8fef-ad40e429f9b4\") " pod="e2e-models-e2e-worker-w0-4d9222/distinct-w0-4d9222-kserve-77444cbdf6-bz5lg"
Sep 18 14:16:54.660949 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:54.660921    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/272988d2-cade-46e9-8fef-ad40e429f9b4-home\") pod \"distinct-w0-4d9222-kserve-77444cbdf6-bz5lg\" (UID: \"272988d2-cade-46e9-8fef-ad40e429f9b4\") " pod="e2e-models-e2e-worker-w0-4d9222/distinct-w0-4d9222-kserve-77444cbdf6-bz5lg"
Sep 18 14:16:54.661086 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:54.661063    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/272988d2-cade-46e9-8fef-ad40e429f9b4-model-cache\") pod \"distinct-w0-4d9222-kserve-77444cbdf6-bz5lg\" (UID: \"272988d2-cade-46e9-8fef-ad40e429f9b4\") " pod="e2e-models-e2e-worker-w0-4d9222/distinct-w0-4d9222-kserve-77444cbdf6-bz5lg"
Sep 18 14:16:54.661152 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:54.661092    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/272988d2-cade-46e9-8fef-ad40e429f9b4-tmp-dir\") pod \"distinct-w0-4d9222-kserve-77444cbdf6-bz5lg\" (UID: \"272988d2-cade-46e9-8fef-ad40e429f9b4\") " pod="e2e-models-e2e-worker-w0-4d9222/distinct-w0-4d9222-kserve-77444cbdf6-bz5lg"
Sep 18 14:16:54.663021 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:54.662995    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/272988d2-cade-46e9-8fef-ad40e429f9b4-dshm\") pod \"distinct-w0-4d9222-kserve-77444cbdf6-bz5lg\" (UID: \"272988d2-cade-46e9-8fef-ad40e429f9b4\") " pod="e2e-models-e2e-worker-w0-4d9222/distinct-w0-4d9222-kserve-77444cbdf6-bz5lg"
Sep 18 14:16:54.663195 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:54.663157    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/272988d2-cade-46e9-8fef-ad40e429f9b4-tls-certs\") pod \"distinct-w0-4d9222-kserve-77444cbdf6-bz5lg\" (UID: \"272988d2-cade-46e9-8fef-ad40e429f9b4\") " pod="e2e-models-e2e-worker-w0-4d9222/distinct-w0-4d9222-kserve-77444cbdf6-bz5lg"
Sep 18 14:16:54.676447 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:54.676429    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfffg\" (UniqueName: \"kubernetes.io/projected/272988d2-cade-46e9-8fef-ad40e429f9b4-kube-api-access-bfffg\") pod \"distinct-w0-4d9222-kserve-77444cbdf6-bz5lg\" (UID: \"272988d2-cade-46e9-8fef-ad40e429f9b4\") " pod="e2e-models-e2e-worker-w0-4d9222/distinct-w0-4d9222-kserve-77444cbdf6-bz5lg"
Sep 18 14:16:54.707184 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:54.707155    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:54.844270 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:54.844233    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4d9222/distinct-w0-4d9222-kserve-77444cbdf6-bz5lg"
Sep 18 14:16:54.974619 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:54.974592    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:55.126736 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:55.126712    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4d9222/distinct-w0-4d9222-kserve-77444cbdf6-bz5lg"]
Sep 18 14:16:55.128800 ip-10-0-143-120 kubenswrapper[2579]: W0918 14:16:55.128774    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod272988d2_cade_46e9_8fef_ad40e429f9b4.slice/crio-49f7f42df7987e6874a859b6bba394097d7578268d865673aad11134846081a6 WatchSource:0}: Error finding container 49f7f42df7987e6874a859b6bba394097d7578268d865673aad11134846081a6: Status 404 returned error can't find the container with id 49f7f42df7987e6874a859b6bba394097d7578268d865673aad11134846081a6
Sep 18 14:16:55.730228 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:55.730191    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4d9222/distinct-w0-4d9222-kserve-77444cbdf6-bz5lg" event={"ID":"272988d2-cade-46e9-8fef-ad40e429f9b4","Type":"ContainerStarted","Data":"ed4e69524692cec35a03e661facaac270865797c3ddbc31d9f01f9b7466ed607"}
Sep 18 14:16:55.730368 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:55.730238    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4d9222/distinct-w0-4d9222-kserve-77444cbdf6-bz5lg" event={"ID":"272988d2-cade-46e9-8fef-ad40e429f9b4","Type":"ContainerStarted","Data":"49f7f42df7987e6874a859b6bba394097d7578268d865673aad11134846081a6"}
Sep 18 14:16:55.751167 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:55.751121    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-4d9222/distinct-w0-4d9222-kserve-77444cbdf6-bz5lg" podStartSLOduration=1.751107609 podStartE2EDuration="1.751107609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:16:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:16:55.748931185 +0000 UTC m=+1087.637647115" watchObservedRunningTime="2026-09-18 14:16:55.751107609 +0000 UTC m=+1087.639823539"
Sep 18 14:16:55.754157 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:55.754124    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:57.222780 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:57.222744    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:57.350822 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:57.350793    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:57.582727 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:57.582691    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:57.697400 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:57.697355    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:58.616454 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:58.616425    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:58.726181 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:58.726148    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:58.911910 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:58.911830    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2"]
Sep 18 14:16:58.912091 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:58.912063    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2" podUID="f7609088-ae6a-40ee-8c95-bcd5e27fa0ce" containerName="istio-proxy" containerID="cri-o://0cd18f8a55e944583bcebe0087ccf613ce44c5b34e9bd52b8f5e617912b05e8e" gracePeriod=30
Sep 18 14:16:58.950287 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:58.950259    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:59.281281 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:59.281204    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:59.419108 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:59.419077    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:16:59.514135 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:16:59.514103    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:02.909185 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:02.909109    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:03.959436 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:03.959397    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:04.172726 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:04.172657    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2"
Sep 18 14:17:04.242148 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:04.242122    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-istio-envoy\") pod \"f7609088-ae6a-40ee-8c95-bcd5e27fa0ce\" (UID: \"f7609088-ae6a-40ee-8c95-bcd5e27fa0ce\") "
Sep 18 14:17:04.242148 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:04.242151    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-istio-token\") pod \"f7609088-ae6a-40ee-8c95-bcd5e27fa0ce\" (UID: \"f7609088-ae6a-40ee-8c95-bcd5e27fa0ce\") "
Sep 18 14:17:04.242352 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:04.242212    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-workload-certs\") pod \"f7609088-ae6a-40ee-8c95-bcd5e27fa0ce\" (UID: \"f7609088-ae6a-40ee-8c95-bcd5e27fa0ce\") "
Sep 18 14:17:04.242352 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:04.242239    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-istio-data\") pod \"f7609088-ae6a-40ee-8c95-bcd5e27fa0ce\" (UID: \"f7609088-ae6a-40ee-8c95-bcd5e27fa0ce\") "
Sep 18 14:17:04.242352 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:04.242284    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-credential-socket\") pod \"f7609088-ae6a-40ee-8c95-bcd5e27fa0ce\" (UID: \"f7609088-ae6a-40ee-8c95-bcd5e27fa0ce\") "
Sep 18 14:17:04.242352 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:04.242324    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6c5q\" (UniqueName: \"kubernetes.io/projected/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-kube-api-access-s6c5q\") pod \"f7609088-ae6a-40ee-8c95-bcd5e27fa0ce\" (UID: \"f7609088-ae6a-40ee-8c95-bcd5e27fa0ce\") "
Sep 18 14:17:04.242532 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:04.242354    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-istiod-ca-cert\") pod \"f7609088-ae6a-40ee-8c95-bcd5e27fa0ce\" (UID: \"f7609088-ae6a-40ee-8c95-bcd5e27fa0ce\") "
Sep 18 14:17:04.242532 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:04.242381    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-workload-socket\") pod \"f7609088-ae6a-40ee-8c95-bcd5e27fa0ce\" (UID: \"f7609088-ae6a-40ee-8c95-bcd5e27fa0ce\") "
Sep 18 14:17:04.242532 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:04.242408    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-istio-podinfo\") pod \"f7609088-ae6a-40ee-8c95-bcd5e27fa0ce\" (UID: \"f7609088-ae6a-40ee-8c95-bcd5e27fa0ce\") "
Sep 18 14:17:04.242532 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:04.242449    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f7609088-ae6a-40ee-8c95-bcd5e27fa0ce" (UID: "f7609088-ae6a-40ee-8c95-bcd5e27fa0ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:17:04.242747 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:04.242591    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-istio-data" (OuterVolumeSpecName: "istio-data") pod "f7609088-ae6a-40ee-8c95-bcd5e27fa0ce" (UID: "f7609088-ae6a-40ee-8c95-bcd5e27fa0ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:17:04.242747 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:04.242603    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f7609088-ae6a-40ee-8c95-bcd5e27fa0ce" (UID: "f7609088-ae6a-40ee-8c95-bcd5e27fa0ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:17:04.242747 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:04.242717    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:17:04.242747 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:04.242734    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:17:04.242747 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:04.242746    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:17:04.242960 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:04.242748    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f7609088-ae6a-40ee-8c95-bcd5e27fa0ce" (UID: "f7609088-ae6a-40ee-8c95-bcd5e27fa0ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:17:04.242960 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:04.242784    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f7609088-ae6a-40ee-8c95-bcd5e27fa0ce" (UID: "f7609088-ae6a-40ee-8c95-bcd5e27fa0ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:17:04.244488 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:04.244465    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-istio-token" (OuterVolumeSpecName: "istio-token") pod "f7609088-ae6a-40ee-8c95-bcd5e27fa0ce" (UID: "f7609088-ae6a-40ee-8c95-bcd5e27fa0ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:17:04.244576 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:04.244554    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f7609088-ae6a-40ee-8c95-bcd5e27fa0ce" (UID: "f7609088-ae6a-40ee-8c95-bcd5e27fa0ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 14:17:04.244747 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:04.244723    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f7609088-ae6a-40ee-8c95-bcd5e27fa0ce" (UID: "f7609088-ae6a-40ee-8c95-bcd5e27fa0ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:17:04.244747 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:04.244733    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-kube-api-access-s6c5q" (OuterVolumeSpecName: "kube-api-access-s6c5q") pod "f7609088-ae6a-40ee-8c95-bcd5e27fa0ce" (UID: "f7609088-ae6a-40ee-8c95-bcd5e27fa0ce"). InnerVolumeSpecName "kube-api-access-s6c5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:17:04.343543 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:04.343514    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6c5q\" (UniqueName: \"kubernetes.io/projected/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-kube-api-access-s6c5q\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:17:04.343543 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:04.343538    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:17:04.343543 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:04.343547    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:17:04.343749 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:04.343556    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:17:04.343749 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:04.343564    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:17:04.343749 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:04.343572    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:17:04.776760 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:04.776723    2579 generic.go:356] "Generic (PLEG): container finished" podID="f7609088-ae6a-40ee-8c95-bcd5e27fa0ce" containerID="0cd18f8a55e944583bcebe0087ccf613ce44c5b34e9bd52b8f5e617912b05e8e" exitCode=0
Sep 18 14:17:04.776917 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:04.776778    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2"
Sep 18 14:17:04.776917 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:04.776795    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2" event={"ID":"f7609088-ae6a-40ee-8c95-bcd5e27fa0ce","Type":"ContainerDied","Data":"0cd18f8a55e944583bcebe0087ccf613ce44c5b34e9bd52b8f5e617912b05e8e"}
Sep 18 14:17:04.776917 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:04.776848    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2" event={"ID":"f7609088-ae6a-40ee-8c95-bcd5e27fa0ce","Type":"ContainerDied","Data":"a0a1d65004e820b184fe2f8ffbc2b4502bf6008f54571dd523e06ff6635e6157"}
Sep 18 14:17:04.776917 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:04.776878    2579 scope.go:117] "RemoveContainer" containerID="0cd18f8a55e944583bcebe0087ccf613ce44c5b34e9bd52b8f5e617912b05e8e"
Sep 18 14:17:04.786720 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:04.786701    2579 scope.go:117] "RemoveContainer" containerID="4d97e0d8ac4508f9002e5ce74ca516cd96434ed15050b31f16ba9d37843521c2"
Sep 18 14:17:04.797347 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:04.797328    2579 scope.go:117] "RemoveContainer" containerID="0cd18f8a55e944583bcebe0087ccf613ce44c5b34e9bd52b8f5e617912b05e8e"
Sep 18 14:17:04.797609 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:17:04.797588    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cd18f8a55e944583bcebe0087ccf613ce44c5b34e9bd52b8f5e617912b05e8e\": container with ID starting with 0cd18f8a55e944583bcebe0087ccf613ce44c5b34e9bd52b8f5e617912b05e8e not found: ID does not exist" containerID="0cd18f8a55e944583bcebe0087ccf613ce44c5b34e9bd52b8f5e617912b05e8e"
Sep 18 14:17:04.797664 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:04.797619    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cd18f8a55e944583bcebe0087ccf613ce44c5b34e9bd52b8f5e617912b05e8e"} err="failed to get container status \"0cd18f8a55e944583bcebe0087ccf613ce44c5b34e9bd52b8f5e617912b05e8e\": rpc error: code = NotFound desc = could not find container \"0cd18f8a55e944583bcebe0087ccf613ce44c5b34e9bd52b8f5e617912b05e8e\": container with ID starting with 0cd18f8a55e944583bcebe0087ccf613ce44c5b34e9bd52b8f5e617912b05e8e not found: ID does not exist"
Sep 18 14:17:04.797664 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:04.797638    2579 scope.go:117] "RemoveContainer" containerID="4d97e0d8ac4508f9002e5ce74ca516cd96434ed15050b31f16ba9d37843521c2"
Sep 18 14:17:04.797871 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:17:04.797852    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d97e0d8ac4508f9002e5ce74ca516cd96434ed15050b31f16ba9d37843521c2\": container with ID starting with 4d97e0d8ac4508f9002e5ce74ca516cd96434ed15050b31f16ba9d37843521c2 not found: ID does not exist" containerID="4d97e0d8ac4508f9002e5ce74ca516cd96434ed15050b31f16ba9d37843521c2"
Sep 18 14:17:04.797930 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:04.797880    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d97e0d8ac4508f9002e5ce74ca516cd96434ed15050b31f16ba9d37843521c2"} err="failed to get container status \"4d97e0d8ac4508f9002e5ce74ca516cd96434ed15050b31f16ba9d37843521c2\": rpc error: code = NotFound desc = could not find container \"4d97e0d8ac4508f9002e5ce74ca516cd96434ed15050b31f16ba9d37843521c2\": container with ID starting with 4d97e0d8ac4508f9002e5ce74ca516cd96434ed15050b31f16ba9d37843521c2 not found: ID does not exist"
Sep 18 14:17:04.801708 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:04.801685    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2"]
Sep 18 14:17:04.805491 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:04.805470    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-834b7e2a-openshift-default-85c77cbb48-lcst2"]
Sep 18 14:17:04.844494 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:04.844475    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-4d9222/distinct-w0-4d9222-kserve-77444cbdf6-bz5lg"
Sep 18 14:17:04.844746 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:04.844728    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4d9222/distinct-w0-4d9222-kserve-77444cbdf6-bz5lg"
Sep 18 14:17:04.857251 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:04.857232    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-4d9222/distinct-w0-4d9222-kserve-77444cbdf6-bz5lg"
Sep 18 14:17:05.793119 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:05.793089    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4d9222/distinct-w0-4d9222-kserve-77444cbdf6-bz5lg"
Sep 18 14:17:06.036451 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:06.036413    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7609088-ae6a-40ee-8c95-bcd5e27fa0ce" path="/var/lib/kubelet/pods/f7609088-ae6a-40ee-8c95-bcd5e27fa0ce/volumes"
Sep 18 14:17:06.256815 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:06.256775    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:06.379458 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:06.379430    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:06.476565 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:06.476537    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:06.602570 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:06.602538    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:13.778240 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:13.778205    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:13.878982 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:13.878957    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:14.188433 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:14.188400    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:14.679583 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:14.679546    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:14.971813 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:14.971711    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm"]
Sep 18 14:17:14.972351 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:14.972328    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7609088-ae6a-40ee-8c95-bcd5e27fa0ce" containerName="istio-proxy"
Sep 18 14:17:14.972351 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:14.972348    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7609088-ae6a-40ee-8c95-bcd5e27fa0ce" containerName="istio-proxy"
Sep 18 14:17:14.972535 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:14.972359    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="458aaf7d-9a1b-4357-beb1-29f71290124d" containerName="istio-proxy"
Sep 18 14:17:14.972535 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:14.972365    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="458aaf7d-9a1b-4357-beb1-29f71290124d" containerName="istio-proxy"
Sep 18 14:17:14.972535 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:14.972372    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7609088-ae6a-40ee-8c95-bcd5e27fa0ce" containerName="istio-proxy"
Sep 18 14:17:14.972535 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:14.972378    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7609088-ae6a-40ee-8c95-bcd5e27fa0ce" containerName="istio-proxy"
Sep 18 14:17:14.972535 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:14.972438    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7609088-ae6a-40ee-8c95-bcd5e27fa0ce" containerName="istio-proxy"
Sep 18 14:17:14.975400 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:14.975384    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm"
Sep 18 14:17:14.977812 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:14.977794    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-526b97-openshift-default-dockercfg-n6vxs\""
Sep 18 14:17:14.990322 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:14.990300    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm"]
Sep 18 14:17:15.001742 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:15.001716    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:15.027427 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:15.027403    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-istio-envoy\") pod \"e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm\" (UID: \"58292587-7ef6-4a85-a3b2-fede8ba8ca4c\") " pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm"
Sep 18 14:17:15.027512 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:15.027439    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-istio-token\") pod \"e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm\" (UID: \"58292587-7ef6-4a85-a3b2-fede8ba8ca4c\") " pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm"
Sep 18 14:17:15.027512 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:15.027470    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-workload-certs\") pod \"e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm\" (UID: \"58292587-7ef6-4a85-a3b2-fede8ba8ca4c\") " pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm"
Sep 18 14:17:15.027590 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:15.027515    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-istiod-ca-cert\") pod \"e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm\" (UID: \"58292587-7ef6-4a85-a3b2-fede8ba8ca4c\") " pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm"
Sep 18 14:17:15.027590 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:15.027543    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-credential-socket\") pod \"e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm\" (UID: \"58292587-7ef6-4a85-a3b2-fede8ba8ca4c\") " pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm"
Sep 18 14:17:15.027590 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:15.027564    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-istio-podinfo\") pod \"e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm\" (UID: \"58292587-7ef6-4a85-a3b2-fede8ba8ca4c\") " pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm"
Sep 18 14:17:15.027686 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:15.027600    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-istio-data\") pod \"e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm\" (UID: \"58292587-7ef6-4a85-a3b2-fede8ba8ca4c\") " pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm"
Sep 18 14:17:15.027686 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:15.027657    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-workload-socket\") pod \"e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm\" (UID: \"58292587-7ef6-4a85-a3b2-fede8ba8ca4c\") " pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm"
Sep 18 14:17:15.027686 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:15.027673    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wl5s\" (UniqueName: \"kubernetes.io/projected/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-kube-api-access-6wl5s\") pod \"e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm\" (UID: \"58292587-7ef6-4a85-a3b2-fede8ba8ca4c\") " pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm"
Sep 18 14:17:15.128679 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:15.128648    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-istio-data\") pod \"e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm\" (UID: \"58292587-7ef6-4a85-a3b2-fede8ba8ca4c\") " pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm"
Sep 18 14:17:15.128845 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:15.128687    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-workload-socket\") pod \"e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm\" (UID: \"58292587-7ef6-4a85-a3b2-fede8ba8ca4c\") " pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm"
Sep 18 14:17:15.128845 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:15.128706    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wl5s\" (UniqueName: \"kubernetes.io/projected/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-kube-api-access-6wl5s\") pod \"e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm\" (UID: \"58292587-7ef6-4a85-a3b2-fede8ba8ca4c\") " pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm"
Sep 18 14:17:15.128845 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:15.128752    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-istio-envoy\") pod \"e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm\" (UID: \"58292587-7ef6-4a85-a3b2-fede8ba8ca4c\") " pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm"
Sep 18 14:17:15.128845 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:15.128790    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-istio-token\") pod \"e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm\" (UID: \"58292587-7ef6-4a85-a3b2-fede8ba8ca4c\") " pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm"
Sep 18 14:17:15.129061 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:15.128928    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-workload-certs\") pod \"e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm\" (UID: \"58292587-7ef6-4a85-a3b2-fede8ba8ca4c\") " pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm"
Sep 18 14:17:15.129061 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:15.128985    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-istiod-ca-cert\") pod \"e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm\" (UID: \"58292587-7ef6-4a85-a3b2-fede8ba8ca4c\") " pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm"
Sep 18 14:17:15.129061 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:15.129035    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-credential-socket\") pod \"e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm\" (UID: \"58292587-7ef6-4a85-a3b2-fede8ba8ca4c\") " pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm"
Sep 18 14:17:15.129242 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:15.129066    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-istio-podinfo\") pod \"e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm\" (UID: \"58292587-7ef6-4a85-a3b2-fede8ba8ca4c\") " pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm"
Sep 18 14:17:15.129242 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:15.129094    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-istio-data\") pod \"e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm\" (UID: \"58292587-7ef6-4a85-a3b2-fede8ba8ca4c\") " pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm"
Sep 18 14:17:15.129242 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:15.129131    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-workload-socket\") pod \"e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm\" (UID: \"58292587-7ef6-4a85-a3b2-fede8ba8ca4c\") " pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm"
Sep 18 14:17:15.129441 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:15.129414    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-workload-certs\") pod \"e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm\" (UID: \"58292587-7ef6-4a85-a3b2-fede8ba8ca4c\") " pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm"
Sep 18 14:17:15.129545 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:15.129510    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-credential-socket\") pod \"e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm\" (UID: \"58292587-7ef6-4a85-a3b2-fede8ba8ca4c\") " pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm"
Sep 18 14:17:15.129749 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:15.129731    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-istiod-ca-cert\") pod \"e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm\" (UID: \"58292587-7ef6-4a85-a3b2-fede8ba8ca4c\") " pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm"
Sep 18 14:17:15.131083 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:15.131058    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-istio-envoy\") pod \"e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm\" (UID: \"58292587-7ef6-4a85-a3b2-fede8ba8ca4c\") " pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm"
Sep 18 14:17:15.131421 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:15.131404    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-istio-podinfo\") pod \"e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm\" (UID: \"58292587-7ef6-4a85-a3b2-fede8ba8ca4c\") " pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm"
Sep 18 14:17:15.138654 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:15.138634    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-istio-token\") pod \"e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm\" (UID: \"58292587-7ef6-4a85-a3b2-fede8ba8ca4c\") " pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm"
Sep 18 14:17:15.138990 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:15.138974    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wl5s\" (UniqueName: \"kubernetes.io/projected/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-kube-api-access-6wl5s\") pod \"e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm\" (UID: \"58292587-7ef6-4a85-a3b2-fede8ba8ca4c\") " pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm"
Sep 18 14:17:15.285080 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:15.285020    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm"
Sep 18 14:17:15.519834 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:15.519798    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:15.569191 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:15.569153    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm"]
Sep 18 14:17:15.570846 ip-10-0-143-120 kubenswrapper[2579]: W0918 14:17:15.570812    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58292587_7ef6_4a85_a3b2_fede8ba8ca4c.slice/crio-eda36fa87d6e3d1827a1e0f25da10625dd58777511e10aed864b87e20aa7138d WatchSource:0}: Error finding container eda36fa87d6e3d1827a1e0f25da10625dd58777511e10aed864b87e20aa7138d: Status 404 returned error can't find the container with id eda36fa87d6e3d1827a1e0f25da10625dd58777511e10aed864b87e20aa7138d
Sep 18 14:17:15.573140 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:15.573111    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:17:15.573224 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:15.573188    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:17:15.573224 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:15.573220    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:17:15.831268 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:15.831186    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm" event={"ID":"58292587-7ef6-4a85-a3b2-fede8ba8ca4c","Type":"ContainerStarted","Data":"fffe684e9482ececbe3ba677486bf3e32c2ff62ffad6d57cbac25c9cd7ddf1ad"}
Sep 18 14:17:15.831268 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:15.831249    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm" event={"ID":"58292587-7ef6-4a85-a3b2-fede8ba8ca4c","Type":"ContainerStarted","Data":"eda36fa87d6e3d1827a1e0f25da10625dd58777511e10aed864b87e20aa7138d"}
Sep 18 14:17:15.852357 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:15.852303    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm" podStartSLOduration=1.852286503 podStartE2EDuration="1.852286503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:17:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:17:15.85147889 +0000 UTC m=+1107.740194821" watchObservedRunningTime="2026-09-18 14:17:15.852286503 +0000 UTC m=+1107.741002433"
Sep 18 14:17:16.286147 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:16.286037    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm"
Sep 18 14:17:16.287900 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:16.287875    2579 patch_prober.go:28] interesting pod/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Sep 18 14:17:16.288026 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:16.287947    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm" podUID="58292587-7ef6-4a85-a3b2-fede8ba8ca4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Sep 18 14:17:17.289524 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:17.289490    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm"
Sep 18 14:17:17.639420 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:17.639388    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:17.756977 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:17.756940    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:17.840892 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:17.840863    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm"
Sep 18 14:17:17.841777 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:17.841754    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm"
Sep 18 14:17:17.861313 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:17.861285    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:18.951929 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:18.951879    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:19.735917 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:19.735874    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:20.639000 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:20.638946    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:21.519585 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:21.519556    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:21.635398 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:21.635365    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:22.158521 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:22.158480    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:22.258831 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:22.258801    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:22.347541 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:22.347514    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:24.210792 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:24.210762    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ee8a08/test-model-ee8a08-kserve-d868b947f-8l6w6"]
Sep 18 14:17:24.242878 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:24.242847    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:24.641178 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:24.641139    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:25.214019 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:25.213988    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:25.771297 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:25.771263    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:26.646677 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:26.646646    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:26.748466 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:26.748438    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:27.515828 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:27.515759    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:28.333457 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:28.333425    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:29.182935 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:29.182901    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:29.617897 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:29.617870    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:30.029673 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:30.029598    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:34.830188 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:34.829961    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2cee-546c4854cc-mvws4"]
Sep 18 14:17:34.830691 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:34.830252    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2cee-546c4854cc-mvws4" podUID="f2669c0c-d001-41b0-be3c-42cb419ab807" containerName="maas-api" containerID="cri-o://c4be97e937a24c275a12dfc3706dabcb39f3cb41ed582287eea5daa2de772677" gracePeriod=30
Sep 18 14:17:34.866341 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:34.866309    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:34.957275 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:34.957227    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:35.030118 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:35.030083    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:35.116555 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:35.116477    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:35.442766 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:35.442686    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:35.566658 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:35.566627    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:36.085278 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:36.085252    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2cee-546c4854cc-mvws4"
Sep 18 14:17:36.207214 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:36.207077    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2669c0c-d001-41b0-be3c-42cb419ab807-metrics-certs\") pod \"f2669c0c-d001-41b0-be3c-42cb419ab807\" (UID: \"f2669c0c-d001-41b0-be3c-42cb419ab807\") "
Sep 18 14:17:36.207214 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:36.207144    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2669c0c-d001-41b0-be3c-42cb419ab807-maas-api-tls\") pod \"f2669c0c-d001-41b0-be3c-42cb419ab807\" (UID: \"f2669c0c-d001-41b0-be3c-42cb419ab807\") "
Sep 18 14:17:36.207461 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:36.207227    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2669c0c-d001-41b0-be3c-42cb419ab807-openshift-service-ca\") pod \"f2669c0c-d001-41b0-be3c-42cb419ab807\" (UID: \"f2669c0c-d001-41b0-be3c-42cb419ab807\") "
Sep 18 14:17:36.207461 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:36.207275    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqtf5\" (UniqueName: \"kubernetes.io/projected/f2669c0c-d001-41b0-be3c-42cb419ab807-kube-api-access-tqtf5\") pod \"f2669c0c-d001-41b0-be3c-42cb419ab807\" (UID: \"f2669c0c-d001-41b0-be3c-42cb419ab807\") "
Sep 18 14:17:36.207578 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:36.207549    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2669c0c-d001-41b0-be3c-42cb419ab807-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f2669c0c-d001-41b0-be3c-42cb419ab807" (UID: "f2669c0c-d001-41b0-be3c-42cb419ab807"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:17:36.209390 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:36.209363    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2669c0c-d001-41b0-be3c-42cb419ab807-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f2669c0c-d001-41b0-be3c-42cb419ab807" (UID: "f2669c0c-d001-41b0-be3c-42cb419ab807"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:17:36.209501 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:36.209395    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2669c0c-d001-41b0-be3c-42cb419ab807-kube-api-access-tqtf5" (OuterVolumeSpecName: "kube-api-access-tqtf5") pod "f2669c0c-d001-41b0-be3c-42cb419ab807" (UID: "f2669c0c-d001-41b0-be3c-42cb419ab807"). InnerVolumeSpecName "kube-api-access-tqtf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:17:36.209501 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:36.209402    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2669c0c-d001-41b0-be3c-42cb419ab807-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f2669c0c-d001-41b0-be3c-42cb419ab807" (UID: "f2669c0c-d001-41b0-be3c-42cb419ab807"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:17:36.308297 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:36.308255    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2669c0c-d001-41b0-be3c-42cb419ab807-metrics-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:17:36.308297 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:36.308287    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2669c0c-d001-41b0-be3c-42cb419ab807-maas-api-tls\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:17:36.308297 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:36.308296    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2669c0c-d001-41b0-be3c-42cb419ab807-openshift-service-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:17:36.308553 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:36.308320    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqtf5\" (UniqueName: \"kubernetes.io/projected/f2669c0c-d001-41b0-be3c-42cb419ab807-kube-api-access-tqtf5\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:17:36.466129 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:36.466039    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:36.933718 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:36.933679    2579 generic.go:356] "Generic (PLEG): container finished" podID="f2669c0c-d001-41b0-be3c-42cb419ab807" containerID="c4be97e937a24c275a12dfc3706dabcb39f3cb41ed582287eea5daa2de772677" exitCode=0
Sep 18 14:17:36.933718 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:36.933723    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2cee-546c4854cc-mvws4"
Sep 18 14:17:36.933919 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:36.933738    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2cee-546c4854cc-mvws4" event={"ID":"f2669c0c-d001-41b0-be3c-42cb419ab807","Type":"ContainerDied","Data":"c4be97e937a24c275a12dfc3706dabcb39f3cb41ed582287eea5daa2de772677"}
Sep 18 14:17:36.933919 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:36.933785    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2cee-546c4854cc-mvws4" event={"ID":"f2669c0c-d001-41b0-be3c-42cb419ab807","Type":"ContainerDied","Data":"e5f70a8b3d4603b04d57235bf159f48b785b8ccbf6044cc18c725640f97d6a8f"}
Sep 18 14:17:36.933919 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:36.933807    2579 scope.go:117] "RemoveContainer" containerID="c4be97e937a24c275a12dfc3706dabcb39f3cb41ed582287eea5daa2de772677"
Sep 18 14:17:36.942931 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:36.942916    2579 scope.go:117] "RemoveContainer" containerID="c4be97e937a24c275a12dfc3706dabcb39f3cb41ed582287eea5daa2de772677"
Sep 18 14:17:36.943197 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:17:36.943155    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4be97e937a24c275a12dfc3706dabcb39f3cb41ed582287eea5daa2de772677\": container with ID starting with c4be97e937a24c275a12dfc3706dabcb39f3cb41ed582287eea5daa2de772677 not found: ID does not exist" containerID="c4be97e937a24c275a12dfc3706dabcb39f3cb41ed582287eea5daa2de772677"
Sep 18 14:17:36.943266 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:36.943209    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4be97e937a24c275a12dfc3706dabcb39f3cb41ed582287eea5daa2de772677"} err="failed to get container status \"c4be97e937a24c275a12dfc3706dabcb39f3cb41ed582287eea5daa2de772677\": rpc error: code = NotFound desc = could not find container \"c4be97e937a24c275a12dfc3706dabcb39f3cb41ed582287eea5daa2de772677\": container with ID starting with c4be97e937a24c275a12dfc3706dabcb39f3cb41ed582287eea5daa2de772677 not found: ID does not exist"
Sep 18 14:17:36.958686 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:36.958662    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2cee-546c4854cc-mvws4"]
Sep 18 14:17:36.968517 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:36.968495    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2cee-546c4854cc-mvws4"]
Sep 18 14:17:37.565921 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:37.565873    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:38.035245 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:38.035146    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2669c0c-d001-41b0-be3c-42cb419ab807" path="/var/lib/kubelet/pods/f2669c0c-d001-41b0-be3c-42cb419ab807/volumes"
Sep 18 14:17:38.656410 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:38.656373    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:39.229797 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:39.229752    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-ee8a08/test-model-ee8a08-kserve-d868b947f-8l6w6" podUID="19670d14-3e75-4660-9c15-3854636aa005" containerName="main" containerID="cri-o://f51f4fe9133f7d9e6baf8fefc341616012d1b0b12f8ab984db0b0d1d55a5e388" gracePeriod=45
Sep 18 14:17:39.281392 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:39.281357    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:39.525587 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:39.525567    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ee8a08/test-model-ee8a08-kserve-d868b947f-8l6w6"
Sep 18 14:17:39.618812 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:39.618778    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:39.637078 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:39.637056    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19670d14-3e75-4660-9c15-3854636aa005-dshm\") pod \"19670d14-3e75-4660-9c15-3854636aa005\" (UID: \"19670d14-3e75-4660-9c15-3854636aa005\") "
Sep 18 14:17:39.637165 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:39.637088    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19670d14-3e75-4660-9c15-3854636aa005-tls-certs\") pod \"19670d14-3e75-4660-9c15-3854636aa005\" (UID: \"19670d14-3e75-4660-9c15-3854636aa005\") "
Sep 18 14:17:39.637165 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:39.637132    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19670d14-3e75-4660-9c15-3854636aa005-home\") pod \"19670d14-3e75-4660-9c15-3854636aa005\" (UID: \"19670d14-3e75-4660-9c15-3854636aa005\") "
Sep 18 14:17:39.637165 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:39.637150    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19670d14-3e75-4660-9c15-3854636aa005-model-cache\") pod \"19670d14-3e75-4660-9c15-3854636aa005\" (UID: \"19670d14-3e75-4660-9c15-3854636aa005\") "
Sep 18 14:17:39.637299 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:39.637181    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqbgc\" (UniqueName: \"kubernetes.io/projected/19670d14-3e75-4660-9c15-3854636aa005-kube-api-access-mqbgc\") pod \"19670d14-3e75-4660-9c15-3854636aa005\" (UID: \"19670d14-3e75-4660-9c15-3854636aa005\") "
Sep 18 14:17:39.637350 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:39.637304    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19670d14-3e75-4660-9c15-3854636aa005-tmp-dir\") pod \"19670d14-3e75-4660-9c15-3854636aa005\" (UID: \"19670d14-3e75-4660-9c15-3854636aa005\") "
Sep 18 14:17:39.637464 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:39.637436    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19670d14-3e75-4660-9c15-3854636aa005-home" (OuterVolumeSpecName: "home") pod "19670d14-3e75-4660-9c15-3854636aa005" (UID: "19670d14-3e75-4660-9c15-3854636aa005"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:17:39.637464 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:39.637452    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19670d14-3e75-4660-9c15-3854636aa005-model-cache" (OuterVolumeSpecName: "model-cache") pod "19670d14-3e75-4660-9c15-3854636aa005" (UID: "19670d14-3e75-4660-9c15-3854636aa005"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:17:39.637603 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:39.637584    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19670d14-3e75-4660-9c15-3854636aa005-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "19670d14-3e75-4660-9c15-3854636aa005" (UID: "19670d14-3e75-4660-9c15-3854636aa005"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:17:39.637670 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:39.637651    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19670d14-3e75-4660-9c15-3854636aa005-home\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:17:39.637709 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:39.637674    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19670d14-3e75-4660-9c15-3854636aa005-model-cache\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:17:39.639427 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:39.639406    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19670d14-3e75-4660-9c15-3854636aa005-dshm" (OuterVolumeSpecName: "dshm") pod "19670d14-3e75-4660-9c15-3854636aa005" (UID: "19670d14-3e75-4660-9c15-3854636aa005"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:17:39.639515 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:39.639452    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19670d14-3e75-4660-9c15-3854636aa005-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "19670d14-3e75-4660-9c15-3854636aa005" (UID: "19670d14-3e75-4660-9c15-3854636aa005"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:17:39.639515 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:39.639468    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19670d14-3e75-4660-9c15-3854636aa005-kube-api-access-mqbgc" (OuterVolumeSpecName: "kube-api-access-mqbgc") pod "19670d14-3e75-4660-9c15-3854636aa005" (UID: "19670d14-3e75-4660-9c15-3854636aa005"). InnerVolumeSpecName "kube-api-access-mqbgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:17:39.738768 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:39.738730    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19670d14-3e75-4660-9c15-3854636aa005-tmp-dir\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:17:39.738768 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:39.738762    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19670d14-3e75-4660-9c15-3854636aa005-dshm\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:17:39.738768 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:39.738774    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19670d14-3e75-4660-9c15-3854636aa005-tls-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:17:39.739245 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:39.738786    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqbgc\" (UniqueName: \"kubernetes.io/projected/19670d14-3e75-4660-9c15-3854636aa005-kube-api-access-mqbgc\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:17:39.960695 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:39.960649    2579 generic.go:356] "Generic (PLEG): container finished" podID="19670d14-3e75-4660-9c15-3854636aa005" containerID="f51f4fe9133f7d9e6baf8fefc341616012d1b0b12f8ab984db0b0d1d55a5e388" exitCode=0
Sep 18 14:17:39.960919 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:39.960741    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ee8a08/test-model-ee8a08-kserve-d868b947f-8l6w6" event={"ID":"19670d14-3e75-4660-9c15-3854636aa005","Type":"ContainerDied","Data":"f51f4fe9133f7d9e6baf8fefc341616012d1b0b12f8ab984db0b0d1d55a5e388"}
Sep 18 14:17:39.960919 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:39.960776    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ee8a08/test-model-ee8a08-kserve-d868b947f-8l6w6" event={"ID":"19670d14-3e75-4660-9c15-3854636aa005","Type":"ContainerDied","Data":"450c29ec37687d2358b863fb34557c00481f37a40536e1fd977e8b879f05eddd"}
Sep 18 14:17:39.960919 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:39.960798    2579 scope.go:117] "RemoveContainer" containerID="f51f4fe9133f7d9e6baf8fefc341616012d1b0b12f8ab984db0b0d1d55a5e388"
Sep 18 14:17:39.961093 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:39.960939    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ee8a08/test-model-ee8a08-kserve-d868b947f-8l6w6"
Sep 18 14:17:39.990034 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:39.990012    2579 scope.go:117] "RemoveContainer" containerID="f51f4fe9133f7d9e6baf8fefc341616012d1b0b12f8ab984db0b0d1d55a5e388"
Sep 18 14:17:39.990428 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:17:39.990402    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f51f4fe9133f7d9e6baf8fefc341616012d1b0b12f8ab984db0b0d1d55a5e388\": container with ID starting with f51f4fe9133f7d9e6baf8fefc341616012d1b0b12f8ab984db0b0d1d55a5e388 not found: ID does not exist" containerID="f51f4fe9133f7d9e6baf8fefc341616012d1b0b12f8ab984db0b0d1d55a5e388"
Sep 18 14:17:39.990548 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:39.990435    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f51f4fe9133f7d9e6baf8fefc341616012d1b0b12f8ab984db0b0d1d55a5e388"} err="failed to get container status \"f51f4fe9133f7d9e6baf8fefc341616012d1b0b12f8ab984db0b0d1d55a5e388\": rpc error: code = NotFound desc = could not find container \"f51f4fe9133f7d9e6baf8fefc341616012d1b0b12f8ab984db0b0d1d55a5e388\": container with ID starting with f51f4fe9133f7d9e6baf8fefc341616012d1b0b12f8ab984db0b0d1d55a5e388 not found: ID does not exist"
Sep 18 14:17:40.001940 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:40.001914    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ee8a08/test-model-ee8a08-kserve-d868b947f-8l6w6"]
Sep 18 14:17:40.011454 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:40.011432    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-ee8a08/test-model-ee8a08-kserve-d868b947f-8l6w6"]
Sep 18 14:17:40.034671 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:40.034650    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19670d14-3e75-4660-9c15-3854636aa005" path="/var/lib/kubelet/pods/19670d14-3e75-4660-9c15-3854636aa005/volumes"
Sep 18 14:17:40.636770 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:40.636739    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:41.699783 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:41.699749    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:42.374202 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:42.374157    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:42.841214 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:42.841156    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:43.048058 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:43.048027    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:43.233349 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:43.233241    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:43.441682 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:43.441649    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst"]
Sep 18 14:17:43.441991 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:43.441979    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2669c0c-d001-41b0-be3c-42cb419ab807" containerName="maas-api"
Sep 18 14:17:43.442041 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:43.441992    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2669c0c-d001-41b0-be3c-42cb419ab807" containerName="maas-api"
Sep 18 14:17:43.442041 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:43.442004    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19670d14-3e75-4660-9c15-3854636aa005" containerName="main"
Sep 18 14:17:43.442041 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:43.442010    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="19670d14-3e75-4660-9c15-3854636aa005" containerName="main"
Sep 18 14:17:43.442270 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:43.442072    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2669c0c-d001-41b0-be3c-42cb419ab807" containerName="maas-api"
Sep 18 14:17:43.442270 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:43.442081    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="19670d14-3e75-4660-9c15-3854636aa005" containerName="main"
Sep 18 14:17:43.442270 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:43.442087    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7609088-ae6a-40ee-8c95-bcd5e27fa0ce" containerName="istio-proxy"
Sep 18 14:17:43.444961 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:43.444945    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst"
Sep 18 14:17:43.447934 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:43.447914    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-216d39-openshift-default-dockercfg-fnx2c\""
Sep 18 14:17:43.471926 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:43.471893    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4978be46-34db-43ce-87e2-a9c3bd6df751-istio-token\") pod \"e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst\" (UID: \"4978be46-34db-43ce-87e2-a9c3bd6df751\") " pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst"
Sep 18 14:17:43.476261 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:43.476236    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4978be46-34db-43ce-87e2-a9c3bd6df751-istio-envoy\") pod \"e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst\" (UID: \"4978be46-34db-43ce-87e2-a9c3bd6df751\") " pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst"
Sep 18 14:17:43.476486 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:43.476467    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4978be46-34db-43ce-87e2-a9c3bd6df751-istio-data\") pod \"e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst\" (UID: \"4978be46-34db-43ce-87e2-a9c3bd6df751\") " pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst"
Sep 18 14:17:43.476623 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:43.476605    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdqqp\" (UniqueName: \"kubernetes.io/projected/4978be46-34db-43ce-87e2-a9c3bd6df751-kube-api-access-zdqqp\") pod \"e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst\" (UID: \"4978be46-34db-43ce-87e2-a9c3bd6df751\") " pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst"
Sep 18 14:17:43.476764 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:43.476750    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4978be46-34db-43ce-87e2-a9c3bd6df751-workload-socket\") pod \"e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst\" (UID: \"4978be46-34db-43ce-87e2-a9c3bd6df751\") " pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst"
Sep 18 14:17:43.476869 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:43.476854    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4978be46-34db-43ce-87e2-a9c3bd6df751-credential-socket\") pod \"e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst\" (UID: \"4978be46-34db-43ce-87e2-a9c3bd6df751\") " pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst"
Sep 18 14:17:43.476973 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:43.476960    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4978be46-34db-43ce-87e2-a9c3bd6df751-istio-podinfo\") pod \"e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst\" (UID: \"4978be46-34db-43ce-87e2-a9c3bd6df751\") " pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst"
Sep 18 14:17:43.477093 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:43.477080    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4978be46-34db-43ce-87e2-a9c3bd6df751-workload-certs\") pod \"e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst\" (UID: \"4978be46-34db-43ce-87e2-a9c3bd6df751\") " pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst"
Sep 18 14:17:43.477226 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:43.477211    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4978be46-34db-43ce-87e2-a9c3bd6df751-istiod-ca-cert\") pod \"e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst\" (UID: \"4978be46-34db-43ce-87e2-a9c3bd6df751\") " pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst"
Sep 18 14:17:43.477465 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:43.475289    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst"]
Sep 18 14:17:43.546365 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:43.546335    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:43.578193 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:43.577890    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4978be46-34db-43ce-87e2-a9c3bd6df751-istio-envoy\") pod \"e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst\" (UID: \"4978be46-34db-43ce-87e2-a9c3bd6df751\") " pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst"
Sep 18 14:17:43.578193 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:43.577954    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4978be46-34db-43ce-87e2-a9c3bd6df751-istio-data\") pod \"e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst\" (UID: \"4978be46-34db-43ce-87e2-a9c3bd6df751\") " pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst"
Sep 18 14:17:43.578193 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:43.577983    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdqqp\" (UniqueName: \"kubernetes.io/projected/4978be46-34db-43ce-87e2-a9c3bd6df751-kube-api-access-zdqqp\") pod \"e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst\" (UID: \"4978be46-34db-43ce-87e2-a9c3bd6df751\") " pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst"
Sep 18 14:17:43.578193 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:43.578036    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4978be46-34db-43ce-87e2-a9c3bd6df751-workload-socket\") pod \"e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst\" (UID: \"4978be46-34db-43ce-87e2-a9c3bd6df751\") " pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst"
Sep 18 14:17:43.578193 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:43.578061    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4978be46-34db-43ce-87e2-a9c3bd6df751-credential-socket\") pod \"e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst\" (UID: \"4978be46-34db-43ce-87e2-a9c3bd6df751\") " pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst"
Sep 18 14:17:43.578193 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:43.578088    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4978be46-34db-43ce-87e2-a9c3bd6df751-istio-podinfo\") pod \"e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst\" (UID: \"4978be46-34db-43ce-87e2-a9c3bd6df751\") " pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst"
Sep 18 14:17:43.578193 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:43.578127    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4978be46-34db-43ce-87e2-a9c3bd6df751-workload-certs\") pod \"e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst\" (UID: \"4978be46-34db-43ce-87e2-a9c3bd6df751\") " pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst"
Sep 18 14:17:43.578193 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:43.578154    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4978be46-34db-43ce-87e2-a9c3bd6df751-istiod-ca-cert\") pod \"e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst\" (UID: \"4978be46-34db-43ce-87e2-a9c3bd6df751\") " pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst"
Sep 18 14:17:43.578757 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:43.578229    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4978be46-34db-43ce-87e2-a9c3bd6df751-istio-token\") pod \"e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst\" (UID: \"4978be46-34db-43ce-87e2-a9c3bd6df751\") " pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst"
Sep 18 14:17:43.582191 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:43.579496    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4978be46-34db-43ce-87e2-a9c3bd6df751-istio-data\") pod \"e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst\" (UID: \"4978be46-34db-43ce-87e2-a9c3bd6df751\") " pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst"
Sep 18 14:17:43.582191 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:43.579751    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4978be46-34db-43ce-87e2-a9c3bd6df751-workload-certs\") pod \"e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst\" (UID: \"4978be46-34db-43ce-87e2-a9c3bd6df751\") " pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst"
Sep 18 14:17:43.582191 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:43.580913    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4978be46-34db-43ce-87e2-a9c3bd6df751-istiod-ca-cert\") pod \"e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst\" (UID: \"4978be46-34db-43ce-87e2-a9c3bd6df751\") " pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst"
Sep 18 14:17:43.582506 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:43.582432    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4978be46-34db-43ce-87e2-a9c3bd6df751-workload-socket\") pod \"e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst\" (UID: \"4978be46-34db-43ce-87e2-a9c3bd6df751\") " pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst"
Sep 18 14:17:43.582506 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:43.582451    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4978be46-34db-43ce-87e2-a9c3bd6df751-credential-socket\") pod \"e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst\" (UID: \"4978be46-34db-43ce-87e2-a9c3bd6df751\") " pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst"
Sep 18 14:17:43.582678 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:43.582647    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4978be46-34db-43ce-87e2-a9c3bd6df751-istio-envoy\") pod \"e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst\" (UID: \"4978be46-34db-43ce-87e2-a9c3bd6df751\") " pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst"
Sep 18 14:17:43.583142 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:43.583117    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4978be46-34db-43ce-87e2-a9c3bd6df751-istio-podinfo\") pod \"e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst\" (UID: \"4978be46-34db-43ce-87e2-a9c3bd6df751\") " pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst"
Sep 18 14:17:43.593849 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:43.593831    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4978be46-34db-43ce-87e2-a9c3bd6df751-istio-token\") pod \"e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst\" (UID: \"4978be46-34db-43ce-87e2-a9c3bd6df751\") " pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst"
Sep 18 14:17:43.611981 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:43.611962    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdqqp\" (UniqueName: \"kubernetes.io/projected/4978be46-34db-43ce-87e2-a9c3bd6df751-kube-api-access-zdqqp\") pod \"e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst\" (UID: \"4978be46-34db-43ce-87e2-a9c3bd6df751\") " pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst"
Sep 18 14:17:43.758102 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:43.758071    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst"
Sep 18 14:17:43.942863 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:43.942830    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:44.058845 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:44.058804    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst"]
Sep 18 14:17:44.083093 ip-10-0-143-120 kubenswrapper[2579]: W0918 14:17:44.083066    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4978be46_34db_43ce_87e2_a9c3bd6df751.slice/crio-e7f09316976e6cb39771f23c7e19a0bb31a33a418ddcc5c3bc57f57cc45f6e8d WatchSource:0}: Error finding container e7f09316976e6cb39771f23c7e19a0bb31a33a418ddcc5c3bc57f57cc45f6e8d: Status 404 returned error can't find the container with id e7f09316976e6cb39771f23c7e19a0bb31a33a418ddcc5c3bc57f57cc45f6e8d
Sep 18 14:17:44.086277 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:44.086227    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:17:44.086378 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:44.086318    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:17:44.086378 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:44.086350    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:17:44.169229 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:44.169200    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:44.708467 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:44.708439    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:44.824774 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:44.824747    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:44.963589 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:44.963511    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh"]
Sep 18 14:17:44.964030 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:44.963761    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh" podUID="aebd26b0-fc86-42ed-b073-15eddb601b91" containerName="istio-proxy" containerID="cri-o://6456a1684d9b6480662f56507987a6d2772291995bdd3aa2b92b32275d950b26" gracePeriod=30
Sep 18 14:17:44.983352 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:44.983331    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:44.995962 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:44.995942    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst" event={"ID":"4978be46-34db-43ce-87e2-a9c3bd6df751","Type":"ContainerStarted","Data":"ef3fce4f25faafd3318637d228f2b2503f313c67e091d4af2e3641d381a8cafa"}
Sep 18 14:17:44.996067 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:44.995981    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst" event={"ID":"4978be46-34db-43ce-87e2-a9c3bd6df751","Type":"ContainerStarted","Data":"e7f09316976e6cb39771f23c7e19a0bb31a33a418ddcc5c3bc57f57cc45f6e8d"}
Sep 18 14:17:45.028357 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:45.028310    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst" podStartSLOduration=2.028292894 podStartE2EDuration="2.028292894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:17:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:17:45.025791718 +0000 UTC m=+1136.914507648" watchObservedRunningTime="2026-09-18 14:17:45.028292894 +0000 UTC m=+1136.917008825"
Sep 18 14:17:45.164582 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:45.164540    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:45.758509 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:45.758465    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst"
Sep 18 14:17:45.764695 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:45.764659    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst"
Sep 18 14:17:46.000617 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:46.000581    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst"
Sep 18 14:17:46.001557 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:46.001534    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst"
Sep 18 14:17:46.162110 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:46.162071    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:47.059852 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:47.059812    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:47.256780 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:47.256735    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:47.503512 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:47.503424    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:47.842155 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:47.842127    2579 patch_prober.go:28] interesting pod/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Sep 18 14:17:47.842308 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:47.842233    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm" podUID="58292587-7ef6-4a85-a3b2-fede8ba8ca4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Sep 18 14:17:48.011481 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:48.011389    2579 generic.go:356] "Generic (PLEG): container finished" podID="58292587-7ef6-4a85-a3b2-fede8ba8ca4c" containerID="fffe684e9482ececbe3ba677486bf3e32c2ff62ffad6d57cbac25c9cd7ddf1ad" exitCode=0
Sep 18 14:17:48.011633 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:48.011468    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm" event={"ID":"58292587-7ef6-4a85-a3b2-fede8ba8ca4c","Type":"ContainerDied","Data":"fffe684e9482ececbe3ba677486bf3e32c2ff62ffad6d57cbac25c9cd7ddf1ad"}
Sep 18 14:17:48.011880 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:48.011864    2579 scope.go:117] "RemoveContainer" containerID="fffe684e9482ececbe3ba677486bf3e32c2ff62ffad6d57cbac25c9cd7ddf1ad"
Sep 18 14:17:48.012906 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:48.012881    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:17:48.012958 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:48.012948    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:17:48.012991 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:48.012979    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:17:48.507881 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:48.507851    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:49.017810 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:49.017774    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm" event={"ID":"58292587-7ef6-4a85-a3b2-fede8ba8ca4c","Type":"ContainerStarted","Data":"7e79e803447af61f33c8f2cffe7fe92c8cbd2f548066e149ee02abca503fa346"}
Sep 18 14:17:49.286027 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:49.285953    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm"
Sep 18 14:17:49.286218 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:49.286196    2579 patch_prober.go:28] interesting pod/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Sep 18 14:17:49.286278 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:49.286247    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm" podUID="58292587-7ef6-4a85-a3b2-fede8ba8ca4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Sep 18 14:17:49.626358 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:49.626325    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:50.246642 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.246589    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh"
Sep 18 14:17:50.285830 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.285795    2579 patch_prober.go:28] interesting pod/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Sep 18 14:17:50.286029 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.285859    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm" podUID="58292587-7ef6-4a85-a3b2-fede8ba8ca4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Sep 18 14:17:50.342161 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.342134    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qgwx4\" (UniqueName: \"kubernetes.io/projected/aebd26b0-fc86-42ed-b073-15eddb601b91-kube-api-access-qgwx4\") pod \"aebd26b0-fc86-42ed-b073-15eddb601b91\" (UID: \"aebd26b0-fc86-42ed-b073-15eddb601b91\") "
Sep 18 14:17:50.342303 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.342191    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aebd26b0-fc86-42ed-b073-15eddb601b91-istiod-ca-cert\") pod \"aebd26b0-fc86-42ed-b073-15eddb601b91\" (UID: \"aebd26b0-fc86-42ed-b073-15eddb601b91\") "
Sep 18 14:17:50.342303 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.342227    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aebd26b0-fc86-42ed-b073-15eddb601b91-workload-socket\") pod \"aebd26b0-fc86-42ed-b073-15eddb601b91\" (UID: \"aebd26b0-fc86-42ed-b073-15eddb601b91\") "
Sep 18 14:17:50.342303 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.342252    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aebd26b0-fc86-42ed-b073-15eddb601b91-istio-envoy\") pod \"aebd26b0-fc86-42ed-b073-15eddb601b91\" (UID: \"aebd26b0-fc86-42ed-b073-15eddb601b91\") "
Sep 18 14:17:50.342303 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.342293    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aebd26b0-fc86-42ed-b073-15eddb601b91-istio-podinfo\") pod \"aebd26b0-fc86-42ed-b073-15eddb601b91\" (UID: \"aebd26b0-fc86-42ed-b073-15eddb601b91\") "
Sep 18 14:17:50.342528 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.342368    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aebd26b0-fc86-42ed-b073-15eddb601b91-credential-socket\") pod \"aebd26b0-fc86-42ed-b073-15eddb601b91\" (UID: \"aebd26b0-fc86-42ed-b073-15eddb601b91\") "
Sep 18 14:17:50.342528 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.342416    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aebd26b0-fc86-42ed-b073-15eddb601b91-workload-certs\") pod \"aebd26b0-fc86-42ed-b073-15eddb601b91\" (UID: \"aebd26b0-fc86-42ed-b073-15eddb601b91\") "
Sep 18 14:17:50.342528 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.342439    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aebd26b0-fc86-42ed-b073-15eddb601b91-istio-data\") pod \"aebd26b0-fc86-42ed-b073-15eddb601b91\" (UID: \"aebd26b0-fc86-42ed-b073-15eddb601b91\") "
Sep 18 14:17:50.342528 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.342463    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aebd26b0-fc86-42ed-b073-15eddb601b91-istio-token\") pod \"aebd26b0-fc86-42ed-b073-15eddb601b91\" (UID: \"aebd26b0-fc86-42ed-b073-15eddb601b91\") "
Sep 18 14:17:50.342756 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.342579    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aebd26b0-fc86-42ed-b073-15eddb601b91-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "aebd26b0-fc86-42ed-b073-15eddb601b91" (UID: "aebd26b0-fc86-42ed-b073-15eddb601b91"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:17:50.342756 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.342721    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aebd26b0-fc86-42ed-b073-15eddb601b91-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "aebd26b0-fc86-42ed-b073-15eddb601b91" (UID: "aebd26b0-fc86-42ed-b073-15eddb601b91"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:17:50.342756 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.342720    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aebd26b0-fc86-42ed-b073-15eddb601b91-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "aebd26b0-fc86-42ed-b073-15eddb601b91" (UID: "aebd26b0-fc86-42ed-b073-15eddb601b91"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:17:50.342904 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.342888    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aebd26b0-fc86-42ed-b073-15eddb601b91-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:17:50.342960 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.342913    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aebd26b0-fc86-42ed-b073-15eddb601b91-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:17:50.342960 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.342918    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aebd26b0-fc86-42ed-b073-15eddb601b91-istio-data" (OuterVolumeSpecName: "istio-data") pod "aebd26b0-fc86-42ed-b073-15eddb601b91" (UID: "aebd26b0-fc86-42ed-b073-15eddb601b91"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:17:50.342960 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.342932    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aebd26b0-fc86-42ed-b073-15eddb601b91-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:17:50.343187 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.343147    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aebd26b0-fc86-42ed-b073-15eddb601b91-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "aebd26b0-fc86-42ed-b073-15eddb601b91" (UID: "aebd26b0-fc86-42ed-b073-15eddb601b91"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:17:50.344375 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.344351    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aebd26b0-fc86-42ed-b073-15eddb601b91-kube-api-access-qgwx4" (OuterVolumeSpecName: "kube-api-access-qgwx4") pod "aebd26b0-fc86-42ed-b073-15eddb601b91" (UID: "aebd26b0-fc86-42ed-b073-15eddb601b91"). InnerVolumeSpecName "kube-api-access-qgwx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:17:50.344500 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.344472    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aebd26b0-fc86-42ed-b073-15eddb601b91-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "aebd26b0-fc86-42ed-b073-15eddb601b91" (UID: "aebd26b0-fc86-42ed-b073-15eddb601b91"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:17:50.344667 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.344642    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aebd26b0-fc86-42ed-b073-15eddb601b91-istio-token" (OuterVolumeSpecName: "istio-token") pod "aebd26b0-fc86-42ed-b073-15eddb601b91" (UID: "aebd26b0-fc86-42ed-b073-15eddb601b91"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:17:50.344667 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.344645    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aebd26b0-fc86-42ed-b073-15eddb601b91-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "aebd26b0-fc86-42ed-b073-15eddb601b91" (UID: "aebd26b0-fc86-42ed-b073-15eddb601b91"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 14:17:50.423722 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.423691    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-216d39-6449979c-28lww"]
Sep 18 14:17:50.424060 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.424047    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aebd26b0-fc86-42ed-b073-15eddb601b91" containerName="istio-proxy"
Sep 18 14:17:50.424105 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.424062    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="aebd26b0-fc86-42ed-b073-15eddb601b91" containerName="istio-proxy"
Sep 18 14:17:50.424141 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.424129    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="aebd26b0-fc86-42ed-b073-15eddb601b91" containerName="istio-proxy"
Sep 18 14:17:50.426264 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.426247    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-216d39-6449979c-28lww"
Sep 18 14:17:50.431511 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.431294    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-216d39-serving-cert\""
Sep 18 14:17:50.443697 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.443677    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aebd26b0-fc86-42ed-b073-15eddb601b91-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:17:50.443697 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.443698    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aebd26b0-fc86-42ed-b073-15eddb601b91-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:17:50.443831 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.443706    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aebd26b0-fc86-42ed-b073-15eddb601b91-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:17:50.443831 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.443714    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgwx4\" (UniqueName: \"kubernetes.io/projected/aebd26b0-fc86-42ed-b073-15eddb601b91-kube-api-access-qgwx4\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:17:50.443831 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.443723    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aebd26b0-fc86-42ed-b073-15eddb601b91-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:17:50.443831 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.443730    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aebd26b0-fc86-42ed-b073-15eddb601b91-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:17:50.447292 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.447267    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-216d39-6449979c-28lww"]
Sep 18 14:17:50.484913 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.484881    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:50.545418 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.545298    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e987aa89-8e53-41ef-9ab3-48bb6d51ca52-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-216d39-6449979c-28lww\" (UID: \"e987aa89-8e53-41ef-9ab3-48bb6d51ca52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-216d39-6449979c-28lww"
Sep 18 14:17:50.545418 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.545409    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n88w\" (UniqueName: \"kubernetes.io/projected/e987aa89-8e53-41ef-9ab3-48bb6d51ca52-kube-api-access-4n88w\") pod \"maas-api-e2e-shared-b-w2-216d39-6449979c-28lww\" (UID: \"e987aa89-8e53-41ef-9ab3-48bb6d51ca52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-216d39-6449979c-28lww"
Sep 18 14:17:50.545639 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.545443    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e987aa89-8e53-41ef-9ab3-48bb6d51ca52-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-216d39-6449979c-28lww\" (UID: \"e987aa89-8e53-41ef-9ab3-48bb6d51ca52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-216d39-6449979c-28lww"
Sep 18 14:17:50.545639 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.545616    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e987aa89-8e53-41ef-9ab3-48bb6d51ca52-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-216d39-6449979c-28lww\" (UID: \"e987aa89-8e53-41ef-9ab3-48bb6d51ca52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-216d39-6449979c-28lww"
Sep 18 14:17:50.575387 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.575255    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:50.646842 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.646816    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n88w\" (UniqueName: \"kubernetes.io/projected/e987aa89-8e53-41ef-9ab3-48bb6d51ca52-kube-api-access-4n88w\") pod \"maas-api-e2e-shared-b-w2-216d39-6449979c-28lww\" (UID: \"e987aa89-8e53-41ef-9ab3-48bb6d51ca52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-216d39-6449979c-28lww"
Sep 18 14:17:50.647269 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.646853    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e987aa89-8e53-41ef-9ab3-48bb6d51ca52-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-216d39-6449979c-28lww\" (UID: \"e987aa89-8e53-41ef-9ab3-48bb6d51ca52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-216d39-6449979c-28lww"
Sep 18 14:17:50.647269 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.646916    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e987aa89-8e53-41ef-9ab3-48bb6d51ca52-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-216d39-6449979c-28lww\" (UID: \"e987aa89-8e53-41ef-9ab3-48bb6d51ca52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-216d39-6449979c-28lww"
Sep 18 14:17:50.647269 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.646956    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e987aa89-8e53-41ef-9ab3-48bb6d51ca52-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-216d39-6449979c-28lww\" (UID: \"e987aa89-8e53-41ef-9ab3-48bb6d51ca52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-216d39-6449979c-28lww"
Sep 18 14:17:50.647700 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.647680    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e987aa89-8e53-41ef-9ab3-48bb6d51ca52-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-216d39-6449979c-28lww\" (UID: \"e987aa89-8e53-41ef-9ab3-48bb6d51ca52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-216d39-6449979c-28lww"
Sep 18 14:17:50.649382 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.649358    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e987aa89-8e53-41ef-9ab3-48bb6d51ca52-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-216d39-6449979c-28lww\" (UID: \"e987aa89-8e53-41ef-9ab3-48bb6d51ca52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-216d39-6449979c-28lww"
Sep 18 14:17:50.649485 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.649429    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e987aa89-8e53-41ef-9ab3-48bb6d51ca52-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-216d39-6449979c-28lww\" (UID: \"e987aa89-8e53-41ef-9ab3-48bb6d51ca52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-216d39-6449979c-28lww"
Sep 18 14:17:50.661855 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.661828    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n88w\" (UniqueName: \"kubernetes.io/projected/e987aa89-8e53-41ef-9ab3-48bb6d51ca52-kube-api-access-4n88w\") pod \"maas-api-e2e-shared-b-w2-216d39-6449979c-28lww\" (UID: \"e987aa89-8e53-41ef-9ab3-48bb6d51ca52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-216d39-6449979c-28lww"
Sep 18 14:17:50.671691 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.671659    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:50.738544 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:50.738515    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-216d39-6449979c-28lww"
Sep 18 14:17:51.030629 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:51.029724    2579 generic.go:356] "Generic (PLEG): container finished" podID="aebd26b0-fc86-42ed-b073-15eddb601b91" containerID="6456a1684d9b6480662f56507987a6d2772291995bdd3aa2b92b32275d950b26" exitCode=0
Sep 18 14:17:51.030629 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:51.029846    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh" event={"ID":"aebd26b0-fc86-42ed-b073-15eddb601b91","Type":"ContainerDied","Data":"6456a1684d9b6480662f56507987a6d2772291995bdd3aa2b92b32275d950b26"}
Sep 18 14:17:51.030629 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:51.029886    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh" event={"ID":"aebd26b0-fc86-42ed-b073-15eddb601b91","Type":"ContainerDied","Data":"21280a457ef4b76582f97b5b44937d52f7afe05487afc135cab3505e952519f3"}
Sep 18 14:17:51.030629 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:51.029918    2579 scope.go:117] "RemoveContainer" containerID="6456a1684d9b6480662f56507987a6d2772291995bdd3aa2b92b32275d950b26"
Sep 18 14:17:51.030629 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:51.030063    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh"
Sep 18 14:17:51.037937 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:51.037912    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-216d39-6449979c-28lww"]
Sep 18 14:17:51.039684 ip-10-0-143-120 kubenswrapper[2579]: W0918 14:17:51.039655    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode987aa89_8e53_41ef_9ab3_48bb6d51ca52.slice/crio-edd5660f17487cd8c03b2af496f3d72d3eaee5a297ba9bc85ebdfe9753f3fdd8 WatchSource:0}: Error finding container edd5660f17487cd8c03b2af496f3d72d3eaee5a297ba9bc85ebdfe9753f3fdd8: Status 404 returned error can't find the container with id edd5660f17487cd8c03b2af496f3d72d3eaee5a297ba9bc85ebdfe9753f3fdd8
Sep 18 14:17:51.042345 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:51.042329    2579 scope.go:117] "RemoveContainer" containerID="a4828280ba2425af30941e863a455d11397546946b1dfd655ff686b2fa931ba4"
Sep 18 14:17:51.051390 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:51.051374    2579 scope.go:117] "RemoveContainer" containerID="6456a1684d9b6480662f56507987a6d2772291995bdd3aa2b92b32275d950b26"
Sep 18 14:17:51.051637 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:17:51.051618    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6456a1684d9b6480662f56507987a6d2772291995bdd3aa2b92b32275d950b26\": container with ID starting with 6456a1684d9b6480662f56507987a6d2772291995bdd3aa2b92b32275d950b26 not found: ID does not exist" containerID="6456a1684d9b6480662f56507987a6d2772291995bdd3aa2b92b32275d950b26"
Sep 18 14:17:51.051692 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:51.051644    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6456a1684d9b6480662f56507987a6d2772291995bdd3aa2b92b32275d950b26"} err="failed to get container status \"6456a1684d9b6480662f56507987a6d2772291995bdd3aa2b92b32275d950b26\": rpc error: code = NotFound desc = could not find container \"6456a1684d9b6480662f56507987a6d2772291995bdd3aa2b92b32275d950b26\": container with ID starting with 6456a1684d9b6480662f56507987a6d2772291995bdd3aa2b92b32275d950b26 not found: ID does not exist"
Sep 18 14:17:51.051692 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:51.051660    2579 scope.go:117] "RemoveContainer" containerID="a4828280ba2425af30941e863a455d11397546946b1dfd655ff686b2fa931ba4"
Sep 18 14:17:51.051919 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:17:51.051902    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4828280ba2425af30941e863a455d11397546946b1dfd655ff686b2fa931ba4\": container with ID starting with a4828280ba2425af30941e863a455d11397546946b1dfd655ff686b2fa931ba4 not found: ID does not exist" containerID="a4828280ba2425af30941e863a455d11397546946b1dfd655ff686b2fa931ba4"
Sep 18 14:17:51.051966 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:51.051925    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4828280ba2425af30941e863a455d11397546946b1dfd655ff686b2fa931ba4"} err="failed to get container status \"a4828280ba2425af30941e863a455d11397546946b1dfd655ff686b2fa931ba4\": rpc error: code = NotFound desc = could not find container \"a4828280ba2425af30941e863a455d11397546946b1dfd655ff686b2fa931ba4\": container with ID starting with a4828280ba2425af30941e863a455d11397546946b1dfd655ff686b2fa931ba4 not found: ID does not exist"
Sep 18 14:17:51.077114 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:51.076322    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh"]
Sep 18 14:17:51.085438 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:51.085401    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:51.085577 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:51.085412    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-dc2cee-openshift-default-867445f5f4-4z6bh"]
Sep 18 14:17:51.166844 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:51.166708    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:51.285567 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:51.285535    2579 patch_prober.go:28] interesting pod/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Sep 18 14:17:51.285722 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:51.285586    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm" podUID="58292587-7ef6-4a85-a3b2-fede8ba8ca4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Sep 18 14:17:51.670741 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:51.670615    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:52.035081 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:52.035044    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aebd26b0-fc86-42ed-b073-15eddb601b91" path="/var/lib/kubelet/pods/aebd26b0-fc86-42ed-b073-15eddb601b91/volumes"
Sep 18 14:17:52.036605 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:52.036581    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-216d39-6449979c-28lww" event={"ID":"e987aa89-8e53-41ef-9ab3-48bb6d51ca52","Type":"ContainerStarted","Data":"ace33f630457bacd6bd4ce248340d6ffb17d7fb81ed1357e217314f8e7aa0203"}
Sep 18 14:17:52.036702 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:52.036634    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-216d39-6449979c-28lww" event={"ID":"e987aa89-8e53-41ef-9ab3-48bb6d51ca52","Type":"ContainerStarted","Data":"edd5660f17487cd8c03b2af496f3d72d3eaee5a297ba9bc85ebdfe9753f3fdd8"}
Sep 18 14:17:52.036702 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:52.036669    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-216d39-6449979c-28lww"
Sep 18 14:17:52.069677 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:52.069621    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-216d39-6449979c-28lww" podStartSLOduration=1.677791284 podStartE2EDuration="2.069599919s" totalImagesPullingTime="391.808635ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:17:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:17:52.054923926 +0000 UTC m=+1143.943639857" watchObservedRunningTime="2026-09-18 14:17:52.069599919 +0000 UTC m=+1143.958315852"
Sep 18 14:17:52.286268 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:52.286160    2579 patch_prober.go:28] interesting pod/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Sep 18 14:17:52.286268 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:52.286247    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm" podUID="58292587-7ef6-4a85-a3b2-fede8ba8ca4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Sep 18 14:17:52.535823 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:52.535789    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:53.286076 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:53.286040    2579 patch_prober.go:28] interesting pod/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Sep 18 14:17:53.286483 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:53.286094    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm" podUID="58292587-7ef6-4a85-a3b2-fede8ba8ca4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Sep 18 14:17:53.446678 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:53.446638    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:54.285832 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:54.285798    2579 patch_prober.go:28] interesting pod/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Sep 18 14:17:54.285984 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:54.285849    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm" podUID="58292587-7ef6-4a85-a3b2-fede8ba8ca4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Sep 18 14:17:54.334421 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:54.334394    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:55.285969 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:55.285924    2579 patch_prober.go:28] interesting pod/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Sep 18 14:17:55.286258 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:55.285992    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm" podUID="58292587-7ef6-4a85-a3b2-fede8ba8ca4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Sep 18 14:17:55.469588 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:55.469550    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:56.070872 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:56.070839    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ee8a08-9bf55585f-sbtvp"]
Sep 18 14:17:56.071081 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:56.071055    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee8a08-9bf55585f-sbtvp" podUID="3358232d-d8be-48ee-8ee4-3e146330e3ec" containerName="maas-api" containerID="cri-o://f031a54f16aa89737fecd1cff95f954fd516fc97316a5dbec0dac9e38ffe26c3" gracePeriod=30
Sep 18 14:17:56.107785 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:56.107753    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:56.181021 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:56.180984    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:56.279625 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:56.279576    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:56.291443 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:56.291409    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm"
Sep 18 14:17:56.291846 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:56.291827    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm"
Sep 18 14:17:56.292681 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:56.292665    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm"
Sep 18 14:17:56.321822 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:56.321758    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee8a08-9bf55585f-sbtvp"
Sep 18 14:17:56.376988 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:56.376956    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:56.403834 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:56.403801    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbfts\" (UniqueName: \"kubernetes.io/projected/3358232d-d8be-48ee-8ee4-3e146330e3ec-kube-api-access-kbfts\") pod \"3358232d-d8be-48ee-8ee4-3e146330e3ec\" (UID: \"3358232d-d8be-48ee-8ee4-3e146330e3ec\") "
Sep 18 14:17:56.404034 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:56.403848    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3358232d-d8be-48ee-8ee4-3e146330e3ec-maas-api-tls\") pod \"3358232d-d8be-48ee-8ee4-3e146330e3ec\" (UID: \"3358232d-d8be-48ee-8ee4-3e146330e3ec\") "
Sep 18 14:17:56.404034 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:56.403903    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3358232d-d8be-48ee-8ee4-3e146330e3ec-openshift-service-ca\") pod \"3358232d-d8be-48ee-8ee4-3e146330e3ec\" (UID: \"3358232d-d8be-48ee-8ee4-3e146330e3ec\") "
Sep 18 14:17:56.404034 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:56.403999    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3358232d-d8be-48ee-8ee4-3e146330e3ec-metrics-certs\") pod \"3358232d-d8be-48ee-8ee4-3e146330e3ec\" (UID: \"3358232d-d8be-48ee-8ee4-3e146330e3ec\") "
Sep 18 14:17:56.404632 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:56.404601    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3358232d-d8be-48ee-8ee4-3e146330e3ec-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3358232d-d8be-48ee-8ee4-3e146330e3ec" (UID: "3358232d-d8be-48ee-8ee4-3e146330e3ec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:17:56.406209 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:56.406168    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3358232d-d8be-48ee-8ee4-3e146330e3ec-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3358232d-d8be-48ee-8ee4-3e146330e3ec" (UID: "3358232d-d8be-48ee-8ee4-3e146330e3ec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:17:56.406501 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:56.406474    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3358232d-d8be-48ee-8ee4-3e146330e3ec-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3358232d-d8be-48ee-8ee4-3e146330e3ec" (UID: "3358232d-d8be-48ee-8ee4-3e146330e3ec"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:17:56.406501 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:56.406493    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3358232d-d8be-48ee-8ee4-3e146330e3ec-kube-api-access-kbfts" (OuterVolumeSpecName: "kube-api-access-kbfts") pod "3358232d-d8be-48ee-8ee4-3e146330e3ec" (UID: "3358232d-d8be-48ee-8ee4-3e146330e3ec"). InnerVolumeSpecName "kube-api-access-kbfts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:17:56.505637 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:56.505597    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3358232d-d8be-48ee-8ee4-3e146330e3ec-metrics-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:17:56.505637 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:56.505635    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbfts\" (UniqueName: \"kubernetes.io/projected/3358232d-d8be-48ee-8ee4-3e146330e3ec-kube-api-access-kbfts\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:17:56.506106 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:56.505650    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3358232d-d8be-48ee-8ee4-3e146330e3ec-maas-api-tls\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:17:56.506106 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:56.505665    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3358232d-d8be-48ee-8ee4-3e146330e3ec-openshift-service-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:17:56.711689 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:56.711605    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:56.812542 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:56.812512    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:57.065563 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:57.065530    2579 generic.go:356] "Generic (PLEG): container finished" podID="3358232d-d8be-48ee-8ee4-3e146330e3ec" containerID="f031a54f16aa89737fecd1cff95f954fd516fc97316a5dbec0dac9e38ffe26c3" exitCode=0
Sep 18 14:17:57.065715 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:57.065573    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee8a08-9bf55585f-sbtvp"
Sep 18 14:17:57.065715 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:57.065606    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee8a08-9bf55585f-sbtvp" event={"ID":"3358232d-d8be-48ee-8ee4-3e146330e3ec","Type":"ContainerDied","Data":"f031a54f16aa89737fecd1cff95f954fd516fc97316a5dbec0dac9e38ffe26c3"}
Sep 18 14:17:57.065715 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:57.065654    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee8a08-9bf55585f-sbtvp" event={"ID":"3358232d-d8be-48ee-8ee4-3e146330e3ec","Type":"ContainerDied","Data":"83f0e9b654103591368b23513c59236ae7b721a061281fe06e9ff4eb22f90d9e"}
Sep 18 14:17:57.065715 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:57.065677    2579 scope.go:117] "RemoveContainer" containerID="f031a54f16aa89737fecd1cff95f954fd516fc97316a5dbec0dac9e38ffe26c3"
Sep 18 14:17:57.075469 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:57.075308    2579 scope.go:117] "RemoveContainer" containerID="f031a54f16aa89737fecd1cff95f954fd516fc97316a5dbec0dac9e38ffe26c3"
Sep 18 14:17:57.075610 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:17:57.075593    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f031a54f16aa89737fecd1cff95f954fd516fc97316a5dbec0dac9e38ffe26c3\": container with ID starting with f031a54f16aa89737fecd1cff95f954fd516fc97316a5dbec0dac9e38ffe26c3 not found: ID does not exist" containerID="f031a54f16aa89737fecd1cff95f954fd516fc97316a5dbec0dac9e38ffe26c3"
Sep 18 14:17:57.075652 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:57.075618    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f031a54f16aa89737fecd1cff95f954fd516fc97316a5dbec0dac9e38ffe26c3"} err="failed to get container status \"f031a54f16aa89737fecd1cff95f954fd516fc97316a5dbec0dac9e38ffe26c3\": rpc error: code = NotFound desc = could not find container \"f031a54f16aa89737fecd1cff95f954fd516fc97316a5dbec0dac9e38ffe26c3\": container with ID starting with f031a54f16aa89737fecd1cff95f954fd516fc97316a5dbec0dac9e38ffe26c3 not found: ID does not exist"
Sep 18 14:17:57.091093 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:57.091070    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ee8a08-9bf55585f-sbtvp"]
Sep 18 14:17:57.094294 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:57.094273    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ee8a08-9bf55585f-sbtvp"]
Sep 18 14:17:57.633099 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:57.633065    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:58.035073 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:58.034999    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3358232d-d8be-48ee-8ee4-3e146330e3ec" path="/var/lib/kubelet/pods/3358232d-d8be-48ee-8ee4-3e146330e3ec/volumes"
Sep 18 14:17:58.047660 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:58.047635    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-216d39-6449979c-28lww"
Sep 18 14:17:58.778517 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:58.778477    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:17:59.609783 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:17:59.609753    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:00.123023 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:00.122993    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:00.637166 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:00.637131    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:01.498016 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:01.497981    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:01.818185 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:01.818147    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:02.409031 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:02.408995    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:03.195051 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:03.195019    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:04.020832 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:04.020801    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:04.135049 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:04.135019    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:04.338948 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:04.338918    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:04.541505 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:04.541473    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:04.846827 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:04.846800    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:05.153391 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:05.153307    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp"]
Sep 18 14:18:05.153795 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:05.153773    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aebd26b0-fc86-42ed-b073-15eddb601b91" containerName="istio-proxy"
Sep 18 14:18:05.153795 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:05.153794    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="aebd26b0-fc86-42ed-b073-15eddb601b91" containerName="istio-proxy"
Sep 18 14:18:05.153973 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:05.153814    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3358232d-d8be-48ee-8ee4-3e146330e3ec" containerName="maas-api"
Sep 18 14:18:05.153973 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:05.153822    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3358232d-d8be-48ee-8ee4-3e146330e3ec" containerName="maas-api"
Sep 18 14:18:05.153973 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:05.153893    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="aebd26b0-fc86-42ed-b073-15eddb601b91" containerName="istio-proxy"
Sep 18 14:18:05.153973 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:05.153907    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3358232d-d8be-48ee-8ee4-3e146330e3ec" containerName="maas-api"
Sep 18 14:18:05.156733 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:05.156712    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp"
Sep 18 14:18:05.161828 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:05.161802    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-ae54af-openshift-default-dockercfg-6hxsx\""
Sep 18 14:18:05.177163 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:05.177136    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp"]
Sep 18 14:18:05.206392 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:05.206368    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:05.288928 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:05.288896    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b95b7806-9984-4565-a427-06f996c82677-istio-envoy\") pod \"e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp\" (UID: \"b95b7806-9984-4565-a427-06f996c82677\") " pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp"
Sep 18 14:18:05.289101 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:05.288955    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b95b7806-9984-4565-a427-06f996c82677-workload-socket\") pod \"e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp\" (UID: \"b95b7806-9984-4565-a427-06f996c82677\") " pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp"
Sep 18 14:18:05.289101 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:05.288977    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b95b7806-9984-4565-a427-06f996c82677-credential-socket\") pod \"e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp\" (UID: \"b95b7806-9984-4565-a427-06f996c82677\") " pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp"
Sep 18 14:18:05.289101 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:05.289004    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b95b7806-9984-4565-a427-06f996c82677-istio-podinfo\") pod \"e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp\" (UID: \"b95b7806-9984-4565-a427-06f996c82677\") " pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp"
Sep 18 14:18:05.289101 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:05.289033    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b95b7806-9984-4565-a427-06f996c82677-istio-token\") pod \"e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp\" (UID: \"b95b7806-9984-4565-a427-06f996c82677\") " pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp"
Sep 18 14:18:05.289101 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:05.289090    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b95b7806-9984-4565-a427-06f996c82677-workload-certs\") pod \"e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp\" (UID: \"b95b7806-9984-4565-a427-06f996c82677\") " pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp"
Sep 18 14:18:05.289367 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:05.289114    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b95b7806-9984-4565-a427-06f996c82677-istiod-ca-cert\") pod \"e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp\" (UID: \"b95b7806-9984-4565-a427-06f996c82677\") " pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp"
Sep 18 14:18:05.289367 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:05.289167    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b95b7806-9984-4565-a427-06f996c82677-istio-data\") pod \"e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp\" (UID: \"b95b7806-9984-4565-a427-06f996c82677\") " pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp"
Sep 18 14:18:05.289367 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:05.289240    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkhk7\" (UniqueName: \"kubernetes.io/projected/b95b7806-9984-4565-a427-06f996c82677-kube-api-access-vkhk7\") pod \"e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp\" (UID: \"b95b7806-9984-4565-a427-06f996c82677\") " pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp"
Sep 18 14:18:05.390706 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:05.390679    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b95b7806-9984-4565-a427-06f996c82677-istio-token\") pod \"e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp\" (UID: \"b95b7806-9984-4565-a427-06f996c82677\") " pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp"
Sep 18 14:18:05.391148 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:05.390734    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b95b7806-9984-4565-a427-06f996c82677-workload-certs\") pod \"e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp\" (UID: \"b95b7806-9984-4565-a427-06f996c82677\") " pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp"
Sep 18 14:18:05.391148 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:05.390767    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b95b7806-9984-4565-a427-06f996c82677-istiod-ca-cert\") pod \"e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp\" (UID: \"b95b7806-9984-4565-a427-06f996c82677\") " pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp"
Sep 18 14:18:05.391148 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:05.390800    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b95b7806-9984-4565-a427-06f996c82677-istio-data\") pod \"e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp\" (UID: \"b95b7806-9984-4565-a427-06f996c82677\") " pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp"
Sep 18 14:18:05.391148 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:05.390827    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkhk7\" (UniqueName: \"kubernetes.io/projected/b95b7806-9984-4565-a427-06f996c82677-kube-api-access-vkhk7\") pod \"e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp\" (UID: \"b95b7806-9984-4565-a427-06f996c82677\") " pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp"
Sep 18 14:18:05.391148 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:05.390869    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b95b7806-9984-4565-a427-06f996c82677-istio-envoy\") pod \"e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp\" (UID: \"b95b7806-9984-4565-a427-06f996c82677\") " pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp"
Sep 18 14:18:05.391148 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:05.390934    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b95b7806-9984-4565-a427-06f996c82677-workload-socket\") pod \"e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp\" (UID: \"b95b7806-9984-4565-a427-06f996c82677\") " pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp"
Sep 18 14:18:05.391148 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:05.390963    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b95b7806-9984-4565-a427-06f996c82677-credential-socket\") pod \"e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp\" (UID: \"b95b7806-9984-4565-a427-06f996c82677\") " pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp"
Sep 18 14:18:05.391148 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:05.390993    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b95b7806-9984-4565-a427-06f996c82677-istio-podinfo\") pod \"e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp\" (UID: \"b95b7806-9984-4565-a427-06f996c82677\") " pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp"
Sep 18 14:18:05.391603 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:05.391390    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b95b7806-9984-4565-a427-06f996c82677-istio-data\") pod \"e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp\" (UID: \"b95b7806-9984-4565-a427-06f996c82677\") " pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp"
Sep 18 14:18:05.391603 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:05.391449    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b95b7806-9984-4565-a427-06f996c82677-workload-socket\") pod \"e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp\" (UID: \"b95b7806-9984-4565-a427-06f996c82677\") " pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp"
Sep 18 14:18:05.391603 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:05.391466    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b95b7806-9984-4565-a427-06f996c82677-credential-socket\") pod \"e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp\" (UID: \"b95b7806-9984-4565-a427-06f996c82677\") " pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp"
Sep 18 14:18:05.391764 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:05.391633    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b95b7806-9984-4565-a427-06f996c82677-workload-certs\") pod \"e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp\" (UID: \"b95b7806-9984-4565-a427-06f996c82677\") " pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp"
Sep 18 14:18:05.391764 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:05.391686    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b95b7806-9984-4565-a427-06f996c82677-istiod-ca-cert\") pod \"e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp\" (UID: \"b95b7806-9984-4565-a427-06f996c82677\") " pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp"
Sep 18 14:18:05.393436 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:05.393415    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b95b7806-9984-4565-a427-06f996c82677-istio-envoy\") pod \"e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp\" (UID: \"b95b7806-9984-4565-a427-06f996c82677\") " pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp"
Sep 18 14:18:05.393773 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:05.393753    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b95b7806-9984-4565-a427-06f996c82677-istio-podinfo\") pod \"e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp\" (UID: \"b95b7806-9984-4565-a427-06f996c82677\") " pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp"
Sep 18 14:18:05.420294 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:05.420240    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b95b7806-9984-4565-a427-06f996c82677-istio-token\") pod \"e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp\" (UID: \"b95b7806-9984-4565-a427-06f996c82677\") " pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp"
Sep 18 14:18:05.420375 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:05.420340    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkhk7\" (UniqueName: \"kubernetes.io/projected/b95b7806-9984-4565-a427-06f996c82677-kube-api-access-vkhk7\") pod \"e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp\" (UID: \"b95b7806-9984-4565-a427-06f996c82677\") " pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp"
Sep 18 14:18:05.468345 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:05.468323    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp"
Sep 18 14:18:05.522922 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:05.522858    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:05.751313 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:05.751289    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp"]
Sep 18 14:18:05.751772 ip-10-0-143-120 kubenswrapper[2579]: W0918 14:18:05.751740    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb95b7806_9984_4565_a427_06f996c82677.slice/crio-7b1d2b5496430a54ed6f26d492e6b0d13b9553a9f2072942b1b8c448b262bd6a WatchSource:0}: Error finding container 7b1d2b5496430a54ed6f26d492e6b0d13b9553a9f2072942b1b8c448b262bd6a: Status 404 returned error can't find the container with id 7b1d2b5496430a54ed6f26d492e6b0d13b9553a9f2072942b1b8c448b262bd6a
Sep 18 14:18:05.754185 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:05.754129    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:18:05.754277 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:05.754247    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:18:05.754346 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:05.754307    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:18:06.112504 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:06.112469    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp" event={"ID":"b95b7806-9984-4565-a427-06f996c82677","Type":"ContainerStarted","Data":"32c4a0eff350c5c7a09ed6b225d23a25ea6c388bdc6427cada95f588b8e19d19"}
Sep 18 14:18:06.112652 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:06.112516    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp" event={"ID":"b95b7806-9984-4565-a427-06f996c82677","Type":"ContainerStarted","Data":"7b1d2b5496430a54ed6f26d492e6b0d13b9553a9f2072942b1b8c448b262bd6a"}
Sep 18 14:18:06.138145 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:06.138090    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp" podStartSLOduration=1.138072384 podStartE2EDuration="1.138072384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:18:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:18:06.13530843 +0000 UTC m=+1158.024024361" watchObservedRunningTime="2026-09-18 14:18:06.138072384 +0000 UTC m=+1158.026788315"
Sep 18 14:18:06.469059 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:06.468987    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp"
Sep 18 14:18:06.474277 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:06.474253    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp"
Sep 18 14:18:07.117722 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:07.117692    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp"
Sep 18 14:18:07.118755 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:07.118735    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp"
Sep 18 14:18:07.363366 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:07.363335    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:08.263452 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:08.263418    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:08.402905 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:08.402871    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:08.923722 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:08.923684    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:10.004394 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:10.004332    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:11.161750 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:11.161705    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:11.583371 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:11.583332    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:11.904612 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:11.904538    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:12.090520 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:12.090488    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:12.195556 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:12.195484    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:13.148013 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:13.147982    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:13.227664 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:13.227631    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:13.362348 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:13.362290    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:13.676202 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:13.676107    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:13.774022 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:13.773966    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:13.782152 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:13.782126    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ae54af-6b864dd554-7kk9c"]
Sep 18 14:18:13.786015 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:13.785994    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ae54af-6b864dd554-7kk9c"
Sep 18 14:18:13.788905 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:13.788887    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-ae54af-dockercfg-dtpq9\""
Sep 18 14:18:13.789048 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:13.788927    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-ae54af\""
Sep 18 14:18:13.801996 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:13.801976    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ae54af-6b864dd554-7kk9c"]
Sep 18 14:18:13.885730 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:13.885695    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b11577e-8cc1-47f2-b262-6c2043dcafb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-ae54af-6b864dd554-7kk9c\" (UID: \"6b11577e-8cc1-47f2-b262-6c2043dcafb0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ae54af-6b864dd554-7kk9c"
Sep 18 14:18:13.886026 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:13.886001    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbgns\" (UniqueName: \"kubernetes.io/projected/6b11577e-8cc1-47f2-b262-6c2043dcafb0-kube-api-access-xbgns\") pod \"payload-pre-processing-e2e-shared-a-w3-ae54af-6b864dd554-7kk9c\" (UID: \"6b11577e-8cc1-47f2-b262-6c2043dcafb0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ae54af-6b864dd554-7kk9c"
Sep 18 14:18:13.987066 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:13.986972    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b11577e-8cc1-47f2-b262-6c2043dcafb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-ae54af-6b864dd554-7kk9c\" (UID: \"6b11577e-8cc1-47f2-b262-6c2043dcafb0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ae54af-6b864dd554-7kk9c"
Sep 18 14:18:13.987066 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:13.987013    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbgns\" (UniqueName: \"kubernetes.io/projected/6b11577e-8cc1-47f2-b262-6c2043dcafb0-kube-api-access-xbgns\") pod \"payload-pre-processing-e2e-shared-a-w3-ae54af-6b864dd554-7kk9c\" (UID: \"6b11577e-8cc1-47f2-b262-6c2043dcafb0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ae54af-6b864dd554-7kk9c"
Sep 18 14:18:13.987768 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:13.987733    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b11577e-8cc1-47f2-b262-6c2043dcafb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-ae54af-6b864dd554-7kk9c\" (UID: \"6b11577e-8cc1-47f2-b262-6c2043dcafb0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ae54af-6b864dd554-7kk9c"
Sep 18 14:18:13.995564 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:13.995539    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbgns\" (UniqueName: \"kubernetes.io/projected/6b11577e-8cc1-47f2-b262-6c2043dcafb0-kube-api-access-xbgns\") pod \"payload-pre-processing-e2e-shared-a-w3-ae54af-6b864dd554-7kk9c\" (UID: \"6b11577e-8cc1-47f2-b262-6c2043dcafb0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ae54af-6b864dd554-7kk9c"
Sep 18 14:18:14.096211 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:14.096183    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ae54af-6b864dd554-7kk9c"
Sep 18 14:18:14.373599 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:14.372674    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ae54af-6b864dd554-7kk9c"]
Sep 18 14:18:14.387793 ip-10-0-143-120 kubenswrapper[2579]: W0918 14:18:14.387760    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b11577e_8cc1_47f2_b262_6c2043dcafb0.slice/crio-290bb963ca206bd7b07cc2dbff323901ede29f19fc4fdfffa2429fb2085d865a WatchSource:0}: Error finding container 290bb963ca206bd7b07cc2dbff323901ede29f19fc4fdfffa2429fb2085d865a: Status 404 returned error can't find the container with id 290bb963ca206bd7b07cc2dbff323901ede29f19fc4fdfffa2429fb2085d865a
Sep 18 14:18:15.050224 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:15.050190    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:15.187915 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:15.187834    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ae54af-6b864dd554-7kk9c" event={"ID":"6b11577e-8cc1-47f2-b262-6c2043dcafb0","Type":"ContainerStarted","Data":"290bb963ca206bd7b07cc2dbff323901ede29f19fc4fdfffa2429fb2085d865a"}
Sep 18 14:18:15.337849 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:15.337710    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:15.546071 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:15.546037    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:16.259833 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:16.259767    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:17.043339 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:17.043295    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 18 14:18:17.043695 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:17.043389    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst" podUID="4978be46-34db-43ce-87e2-a9c3bd6df751" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 18 14:18:17.201246 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:17.201156    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ae54af-6b864dd554-7kk9c" event={"ID":"6b11577e-8cc1-47f2-b262-6c2043dcafb0","Type":"ContainerStarted","Data":"328d484f2ba3760706836a4a5044146d2295bf3c8ea185a6c71cee9d3c99f83e"}
Sep 18 14:18:17.201396 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:17.201306    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ae54af-6b864dd554-7kk9c"
Sep 18 14:18:17.237633 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:17.237588    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ae54af-6b864dd554-7kk9c" podStartSLOduration=1.7808522230000001 podStartE2EDuration="4.237574756s" totalImagesPullingTime="2.456722533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:18:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:18:17.235051938 +0000 UTC m=+1169.123767870" watchObservedRunningTime="2026-09-18 14:18:17.237574756 +0000 UTC m=+1169.126290707"
Sep 18 14:18:17.285859 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:17.285833    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:18.121741 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:18.121704    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:19.044550 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:19.044523    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:19.212971 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:19.212943    2579 generic.go:356] "Generic (PLEG): container finished" podID="4978be46-34db-43ce-87e2-a9c3bd6df751" containerID="ef3fce4f25faafd3318637d228f2b2503f313c67e091d4af2e3641d381a8cafa" exitCode=0
Sep 18 14:18:19.213340 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:19.212976    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst" event={"ID":"4978be46-34db-43ce-87e2-a9c3bd6df751","Type":"ContainerDied","Data":"ef3fce4f25faafd3318637d228f2b2503f313c67e091d4af2e3641d381a8cafa"}
Sep 18 14:18:19.213387 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:19.213349    2579 scope.go:117] "RemoveContainer" containerID="ef3fce4f25faafd3318637d228f2b2503f313c67e091d4af2e3641d381a8cafa"
Sep 18 14:18:19.214643 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:19.214609    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:18:19.214717 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:19.214677    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:18:19.214717 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:19.214707    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:18:19.784271 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:19.784234    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:20.219507 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:20.219475    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst" event={"ID":"4978be46-34db-43ce-87e2-a9c3bd6df751","Type":"ContainerStarted","Data":"897d743f2e731560531a67d83ffcfb8f80f8b1e859a332c3fce92116987b1938"}
Sep 18 14:18:20.698444 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:20.698405    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-988c9d38-6cc7984df-7hlvf"]
Sep 18 14:18:20.700871 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:20.700849    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-988c9d38-6cc7984df-7hlvf"
Sep 18 14:18:20.703309 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:20.703289    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-988c9d38-serving-cert\""
Sep 18 14:18:20.715145 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:20.715123    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-988c9d38-6cc7984df-7hlvf"]
Sep 18 14:18:20.744205 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:20.744163    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:20.764191 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:20.757597    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b-openshift-service-ca\") pod \"maas-api-e2e-mt-988c9d38-6cc7984df-7hlvf\" (UID: \"26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-988c9d38-6cc7984df-7hlvf"
Sep 18 14:18:20.764191 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:20.757693    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hd77\" (UniqueName: \"kubernetes.io/projected/26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b-kube-api-access-6hd77\") pod \"maas-api-e2e-mt-988c9d38-6cc7984df-7hlvf\" (UID: \"26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-988c9d38-6cc7984df-7hlvf"
Sep 18 14:18:20.764191 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:20.757768    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b-maas-api-tls\") pod \"maas-api-e2e-mt-988c9d38-6cc7984df-7hlvf\" (UID: \"26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-988c9d38-6cc7984df-7hlvf"
Sep 18 14:18:20.764191 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:20.757799    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b-metrics-certs\") pod \"maas-api-e2e-mt-988c9d38-6cc7984df-7hlvf\" (UID: \"26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-988c9d38-6cc7984df-7hlvf"
Sep 18 14:18:20.768191 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:20.764490    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst"
Sep 18 14:18:20.768191 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:20.764591    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 18 14:18:20.768191 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:20.764631    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst" podUID="4978be46-34db-43ce-87e2-a9c3bd6df751" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 18 14:18:20.814577 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:20.814547    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:20.859068 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:20.859039    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b-maas-api-tls\") pod \"maas-api-e2e-mt-988c9d38-6cc7984df-7hlvf\" (UID: \"26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-988c9d38-6cc7984df-7hlvf"
Sep 18 14:18:20.859332 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:18:20.859306    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-988c9d38-serving-cert: secret "maas-api-e2e-mt-988c9d38-serving-cert" not found
Sep 18 14:18:20.859552 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:18:20.859538    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b-maas-api-tls podName:26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b nodeName:}" failed. No retries permitted until 2026-09-18 14:18:21.359512309 +0000 UTC m=+1173.248228241 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b-maas-api-tls") pod "maas-api-e2e-mt-988c9d38-6cc7984df-7hlvf" (UID: "26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b") : secret "maas-api-e2e-mt-988c9d38-serving-cert" not found
Sep 18 14:18:20.859718 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:20.859314    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b-metrics-certs\") pod \"maas-api-e2e-mt-988c9d38-6cc7984df-7hlvf\" (UID: \"26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-988c9d38-6cc7984df-7hlvf"
Sep 18 14:18:20.859959 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:20.859940    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b-openshift-service-ca\") pod \"maas-api-e2e-mt-988c9d38-6cc7984df-7hlvf\" (UID: \"26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-988c9d38-6cc7984df-7hlvf"
Sep 18 14:18:20.860865 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:20.860761    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b-openshift-service-ca\") pod \"maas-api-e2e-mt-988c9d38-6cc7984df-7hlvf\" (UID: \"26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-988c9d38-6cc7984df-7hlvf"
Sep 18 14:18:20.861112 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:20.861091    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hd77\" (UniqueName: \"kubernetes.io/projected/26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b-kube-api-access-6hd77\") pod \"maas-api-e2e-mt-988c9d38-6cc7984df-7hlvf\" (UID: \"26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-988c9d38-6cc7984df-7hlvf"
Sep 18 14:18:20.874219 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:20.874194    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b-metrics-certs\") pod \"maas-api-e2e-mt-988c9d38-6cc7984df-7hlvf\" (UID: \"26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-988c9d38-6cc7984df-7hlvf"
Sep 18 14:18:20.876648 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:20.876626    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hd77\" (UniqueName: \"kubernetes.io/projected/26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b-kube-api-access-6hd77\") pod \"maas-api-e2e-mt-988c9d38-6cc7984df-7hlvf\" (UID: \"26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-988c9d38-6cc7984df-7hlvf"
Sep 18 14:18:21.341614 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:21.341581    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:21.376734 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:21.376691    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b-maas-api-tls\") pod \"maas-api-e2e-mt-988c9d38-6cc7984df-7hlvf\" (UID: \"26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-988c9d38-6cc7984df-7hlvf"
Sep 18 14:18:21.384621 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:21.384597    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b-maas-api-tls\") pod \"maas-api-e2e-mt-988c9d38-6cc7984df-7hlvf\" (UID: \"26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-988c9d38-6cc7984df-7hlvf"
Sep 18 14:18:21.485114 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:21.485081    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:21.612023 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:21.611950    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-988c9d38-6cc7984df-7hlvf"
Sep 18 14:18:21.761993 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:21.761528    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 18 14:18:21.761993 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:21.761602    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst" podUID="4978be46-34db-43ce-87e2-a9c3bd6df751" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 18 14:18:21.929235 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:21.929207    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-988c9d38-6cc7984df-7hlvf"]
Sep 18 14:18:21.931312 ip-10-0-143-120 kubenswrapper[2579]: W0918 14:18:21.931287    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26dc2ae3_49a7_4fc1_b05e_f88d3dbe1d4b.slice/crio-c87cb570750e529f36275282431187494cf2bb1a3e268ac5f96cf5cf09dc2ad8 WatchSource:0}: Error finding container c87cb570750e529f36275282431187494cf2bb1a3e268ac5f96cf5cf09dc2ad8: Status 404 returned error can't find the container with id c87cb570750e529f36275282431187494cf2bb1a3e268ac5f96cf5cf09dc2ad8
Sep 18 14:18:22.233240 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:22.233214    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-988c9d38-6cc7984df-7hlvf" event={"ID":"26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b","Type":"ContainerStarted","Data":"c87cb570750e529f36275282431187494cf2bb1a3e268ac5f96cf5cf09dc2ad8"}
Sep 18 14:18:22.758849 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:22.758807    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 18 14:18:22.759353 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:22.758873    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst" podUID="4978be46-34db-43ce-87e2-a9c3bd6df751" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 18 14:18:23.238908 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:23.238871    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-988c9d38-6cc7984df-7hlvf" event={"ID":"26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b","Type":"ContainerStarted","Data":"5201ec87cc904bfcf469291b76db28dc67dca2245e10aab7b1b758201150101d"}
Sep 18 14:18:23.239080 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:23.238977    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-988c9d38-6cc7984df-7hlvf"
Sep 18 14:18:23.759228 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:23.759191    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 18 14:18:23.759796 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:23.759764    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst" podUID="4978be46-34db-43ce-87e2-a9c3bd6df751" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 18 14:18:24.759388 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:24.759346    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 18 14:18:24.759879 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:24.759413    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst" podUID="4978be46-34db-43ce-87e2-a9c3bd6df751" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 18 14:18:25.762452 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:25.762423    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst"
Sep 18 14:18:25.762869 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:25.762813    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst"
Sep 18 14:18:25.763562 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:25.763541    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst"
Sep 18 14:18:25.788465 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:25.788414    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-988c9d38-6cc7984df-7hlvf" podStartSLOduration=5.520850477 podStartE2EDuration="5.788395748s" totalImagesPullingTime="267.545271ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:18:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:18:23.262482225 +0000 UTC m=+1175.151198157" watchObservedRunningTime="2026-09-18 14:18:25.788395748 +0000 UTC m=+1177.677111680"
Sep 18 14:18:28.208716 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:28.208690    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ae54af-6b864dd554-7kk9c"
Sep 18 14:18:29.250924 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:29.250894    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-988c9d38-6cc7984df-7hlvf"
Sep 18 14:18:34.271913 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:34.271743    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:35.676518 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:35.676484    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:36.468058 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:36.468026    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:37.609786 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:37.609754    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:38.124678 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:38.124648    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:38.159342 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:38.159299    2579 patch_prober.go:28] interesting pod/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 18 14:18:38.159542 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:38.159372    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp" podUID="b95b7806-9984-4565-a427-06f996c82677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 18 14:18:38.555627 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:38.555597    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:38.760435 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:38.760403    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:38.834327 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:38.834249    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn"]
Sep 18 14:18:38.836449 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:38.836435    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn"
Sep 18 14:18:38.839919 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:38.839898    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-980661-openshift-default-dockercfg-j7wpf\""
Sep 18 14:18:38.874215 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:38.874193    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn"]
Sep 18 14:18:38.952012 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:38.951979    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwtmf\" (UniqueName: \"kubernetes.io/projected/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-kube-api-access-zwtmf\") pod \"e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn\" (UID: \"bec5d7ac-13a5-4317-b941-c8c47dbcb25e\") " pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn"
Sep 18 14:18:38.952214 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:38.952025    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-credential-socket\") pod \"e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn\" (UID: \"bec5d7ac-13a5-4317-b941-c8c47dbcb25e\") " pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn"
Sep 18 14:18:38.952214 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:38.952054    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-workload-socket\") pod \"e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn\" (UID: \"bec5d7ac-13a5-4317-b941-c8c47dbcb25e\") " pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn"
Sep 18 14:18:38.952214 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:38.952070    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-workload-certs\") pod \"e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn\" (UID: \"bec5d7ac-13a5-4317-b941-c8c47dbcb25e\") " pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn"
Sep 18 14:18:38.952214 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:38.952091    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-istio-envoy\") pod \"e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn\" (UID: \"bec5d7ac-13a5-4317-b941-c8c47dbcb25e\") " pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn"
Sep 18 14:18:38.952214 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:38.952117    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-istiod-ca-cert\") pod \"e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn\" (UID: \"bec5d7ac-13a5-4317-b941-c8c47dbcb25e\") " pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn"
Sep 18 14:18:38.952214 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:38.952143    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-istio-podinfo\") pod \"e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn\" (UID: \"bec5d7ac-13a5-4317-b941-c8c47dbcb25e\") " pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn"
Sep 18 14:18:38.952214 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:38.952189    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-istio-token\") pod \"e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn\" (UID: \"bec5d7ac-13a5-4317-b941-c8c47dbcb25e\") " pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn"
Sep 18 14:18:38.952214 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:38.952214    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-istio-data\") pod \"e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn\" (UID: \"bec5d7ac-13a5-4317-b941-c8c47dbcb25e\") " pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn"
Sep 18 14:18:38.955604 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:38.955572    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:39.053349 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:39.053323    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwtmf\" (UniqueName: \"kubernetes.io/projected/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-kube-api-access-zwtmf\") pod \"e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn\" (UID: \"bec5d7ac-13a5-4317-b941-c8c47dbcb25e\") " pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn"
Sep 18 14:18:39.053484 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:39.053377    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-credential-socket\") pod \"e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn\" (UID: \"bec5d7ac-13a5-4317-b941-c8c47dbcb25e\") " pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn"
Sep 18 14:18:39.053484 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:39.053422    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-workload-socket\") pod \"e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn\" (UID: \"bec5d7ac-13a5-4317-b941-c8c47dbcb25e\") " pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn"
Sep 18 14:18:39.053484 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:39.053447    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-workload-certs\") pod \"e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn\" (UID: \"bec5d7ac-13a5-4317-b941-c8c47dbcb25e\") " pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn"
Sep 18 14:18:39.053484 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:39.053481    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-istio-envoy\") pod \"e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn\" (UID: \"bec5d7ac-13a5-4317-b941-c8c47dbcb25e\") " pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn"
Sep 18 14:18:39.053709 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:39.053537    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-istiod-ca-cert\") pod \"e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn\" (UID: \"bec5d7ac-13a5-4317-b941-c8c47dbcb25e\") " pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn"
Sep 18 14:18:39.053709 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:39.053580    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-istio-podinfo\") pod \"e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn\" (UID: \"bec5d7ac-13a5-4317-b941-c8c47dbcb25e\") " pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn"
Sep 18 14:18:39.053709 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:39.053606    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-istio-token\") pod \"e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn\" (UID: \"bec5d7ac-13a5-4317-b941-c8c47dbcb25e\") " pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn"
Sep 18 14:18:39.053709 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:39.053648    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-istio-data\") pod \"e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn\" (UID: \"bec5d7ac-13a5-4317-b941-c8c47dbcb25e\") " pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn"
Sep 18 14:18:39.053974 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:39.053862    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-workload-socket\") pod \"e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn\" (UID: \"bec5d7ac-13a5-4317-b941-c8c47dbcb25e\") " pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn"
Sep 18 14:18:39.053974 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:39.053871    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-credential-socket\") pod \"e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn\" (UID: \"bec5d7ac-13a5-4317-b941-c8c47dbcb25e\") " pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn"
Sep 18 14:18:39.054090 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:39.053985    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-istio-data\") pod \"e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn\" (UID: \"bec5d7ac-13a5-4317-b941-c8c47dbcb25e\") " pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn"
Sep 18 14:18:39.054164 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:39.054122    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-workload-certs\") pod \"e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn\" (UID: \"bec5d7ac-13a5-4317-b941-c8c47dbcb25e\") " pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn"
Sep 18 14:18:39.054440 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:39.054409    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-istiod-ca-cert\") pod \"e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn\" (UID: \"bec5d7ac-13a5-4317-b941-c8c47dbcb25e\") " pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn"
Sep 18 14:18:39.056390 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:39.056368    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-istio-podinfo\") pod \"e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn\" (UID: \"bec5d7ac-13a5-4317-b941-c8c47dbcb25e\") " pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn"
Sep 18 14:18:39.056483 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:39.056403    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-istio-envoy\") pod \"e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn\" (UID: \"bec5d7ac-13a5-4317-b941-c8c47dbcb25e\") " pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn"
Sep 18 14:18:39.062395 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:39.062369    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-istio-token\") pod \"e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn\" (UID: \"bec5d7ac-13a5-4317-b941-c8c47dbcb25e\") " pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn"
Sep 18 14:18:39.062761 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:39.062738    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwtmf\" (UniqueName: \"kubernetes.io/projected/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-kube-api-access-zwtmf\") pod \"e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn\" (UID: \"bec5d7ac-13a5-4317-b941-c8c47dbcb25e\") " pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn"
Sep 18 14:18:39.146512 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:39.146461    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn"
Sep 18 14:18:39.178724 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:39.178692    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:39.326331 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:39.326299    2579 generic.go:356] "Generic (PLEG): container finished" podID="b95b7806-9984-4565-a427-06f996c82677" containerID="32c4a0eff350c5c7a09ed6b225d23a25ea6c388bdc6427cada95f588b8e19d19" exitCode=0
Sep 18 14:18:39.326550 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:39.326379    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp" event={"ID":"b95b7806-9984-4565-a427-06f996c82677","Type":"ContainerDied","Data":"32c4a0eff350c5c7a09ed6b225d23a25ea6c388bdc6427cada95f588b8e19d19"}
Sep 18 14:18:39.326866 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:39.326849    2579 scope.go:117] "RemoveContainer" containerID="32c4a0eff350c5c7a09ed6b225d23a25ea6c388bdc6427cada95f588b8e19d19"
Sep 18 14:18:39.328012 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:39.327984    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:18:39.328111 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:39.328070    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:18:39.328198 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:39.328120    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:18:39.442802 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:39.442768    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn"]
Sep 18 14:18:39.443360 ip-10-0-143-120 kubenswrapper[2579]: W0918 14:18:39.443335    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbec5d7ac_13a5_4317_b941_c8c47dbcb25e.slice/crio-22d61d011891b543818332135cb7b9d5036ae72e491cd604719b8c2eb856c56b WatchSource:0}: Error finding container 22d61d011891b543818332135cb7b9d5036ae72e491cd604719b8c2eb856c56b: Status 404 returned error can't find the container with id 22d61d011891b543818332135cb7b9d5036ae72e491cd604719b8c2eb856c56b
Sep 18 14:18:39.445914 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:39.445876    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:18:39.446021 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:39.445964    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:18:39.446095 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:39.446024    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:18:39.806029 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:39.805994    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:40.130498 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:40.130414    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:40.332342 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:40.332296    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn" event={"ID":"bec5d7ac-13a5-4317-b941-c8c47dbcb25e","Type":"ContainerStarted","Data":"0141d3ef082394e73b9a341213e8f15d3f9795aa96d23ee0952356a4f0bb04bd"}
Sep 18 14:18:40.332534 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:40.332360    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn" event={"ID":"bec5d7ac-13a5-4317-b941-c8c47dbcb25e","Type":"ContainerStarted","Data":"22d61d011891b543818332135cb7b9d5036ae72e491cd604719b8c2eb856c56b"}
Sep 18 14:18:40.334293 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:40.334269    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp" event={"ID":"b95b7806-9984-4565-a427-06f996c82677","Type":"ContainerStarted","Data":"c0fc8fe4b2e9498b02be786d7548807b0ae4f352c3c7ebc6e1b4f673617e5796"}
Sep 18 14:18:40.357972 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:40.357922    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn" podStartSLOduration=2.357904433 podStartE2EDuration="2.357904433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:18:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:18:40.357712337 +0000 UTC m=+1192.246428264" watchObservedRunningTime="2026-09-18 14:18:40.357904433 +0000 UTC m=+1192.246620365"
Sep 18 14:18:40.468833 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:40.468754    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp"
Sep 18 14:18:40.468976 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:40.468939    2579 patch_prober.go:28] interesting pod/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 18 14:18:40.469031 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:40.468987    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp" podUID="b95b7806-9984-4565-a427-06f996c82677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 18 14:18:40.795075 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:40.794982    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:41.146813 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:41.146783    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn"
Sep 18 14:18:41.152352 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:41.152324    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn"
Sep 18 14:18:41.339215 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:41.339184    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn"
Sep 18 14:18:41.340248 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:41.340229    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn"
Sep 18 14:18:41.477664 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:41.477560    2579 patch_prober.go:28] interesting pod/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 18 14:18:41.477664 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:41.477646    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp" podUID="b95b7806-9984-4565-a427-06f996c82677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 18 14:18:41.977712 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:41.977676    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:42.385652 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:42.385616    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:42.469200 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:42.469143    2579 patch_prober.go:28] interesting pod/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 18 14:18:42.469493 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:42.469465    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp" podUID="b95b7806-9984-4565-a427-06f996c82677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 18 14:18:42.568026 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:42.567994    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:43.135117 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:43.135072    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:43.469678 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:43.469589    2579 patch_prober.go:28] interesting pod/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 18 14:18:43.470156 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:43.469673    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp" podUID="b95b7806-9984-4565-a427-06f996c82677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 18 14:18:43.931638 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:43.931608    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:44.468943 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:44.468907    2579 patch_prober.go:28] interesting pod/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 18 14:18:44.469251 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:44.469220    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp" podUID="b95b7806-9984-4565-a427-06f996c82677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 18 14:18:44.741650 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:44.741573    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:44.841623 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:44.841591    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:45.469524 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:45.469487    2579 patch_prober.go:28] interesting pod/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 18 14:18:45.469809 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:45.469782    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp" podUID="b95b7806-9984-4565-a427-06f996c82677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 18 14:18:45.878573 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:45.878534    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:46.472529 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:46.472503    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp"
Sep 18 14:18:46.472895 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:46.472873    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp"
Sep 18 14:18:46.473676 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:46.473651    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp"
Sep 18 14:18:46.807422 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:46.807397    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:47.845961 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:47.845924    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:47.962773 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:47.962727    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:48.279625 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:48.279539    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:48.565497 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:48.565466    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:48.680641 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:48.680604    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:48.708637 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:48.708600    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-980661-77765769cc-rvgpd"]
Sep 18 14:18:48.711031 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:48.711012    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-980661-77765769cc-rvgpd"
Sep 18 14:18:48.714928 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:48.714907    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-980661\""
Sep 18 14:18:48.714928 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:48.714922    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-980661-dockercfg-qqjh7\""
Sep 18 14:18:48.727606 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:48.727580    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-980661-77765769cc-rvgpd"]
Sep 18 14:18:48.742813 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:48.742783    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4hh7\" (UniqueName: \"kubernetes.io/projected/0e570098-8720-40dd-944b-28607af93d0a-kube-api-access-b4hh7\") pod \"payload-processing-e2e-shared-b-w3-980661-77765769cc-rvgpd\" (UID: \"0e570098-8720-40dd-944b-28607af93d0a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-980661-77765769cc-rvgpd"
Sep 18 14:18:48.742968 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:48.742889    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e570098-8720-40dd-944b-28607af93d0a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-980661-77765769cc-rvgpd\" (UID: \"0e570098-8720-40dd-944b-28607af93d0a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-980661-77765769cc-rvgpd"
Sep 18 14:18:48.746592 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:48.746567    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-980661-68db9c5866-5hkmc"]
Sep 18 14:18:48.748873 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:48.748855    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-980661-68db9c5866-5hkmc"
Sep 18 14:18:48.765390 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:48.765364    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-980661-68db9c5866-5hkmc"]
Sep 18 14:18:48.844260 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:48.844163    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e570098-8720-40dd-944b-28607af93d0a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-980661-77765769cc-rvgpd\" (UID: \"0e570098-8720-40dd-944b-28607af93d0a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-980661-77765769cc-rvgpd"
Sep 18 14:18:48.844260 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:48.844226    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6rcc\" (UniqueName: \"kubernetes.io/projected/f65a88fe-acd7-4b42-808f-81f80c051a3e-kube-api-access-h6rcc\") pod \"payload-pre-processing-e2e-shared-b-w3-980661-68db9c5866-5hkmc\" (UID: \"f65a88fe-acd7-4b42-808f-81f80c051a3e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-980661-68db9c5866-5hkmc"
Sep 18 14:18:48.844438 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:48.844272    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f65a88fe-acd7-4b42-808f-81f80c051a3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-980661-68db9c5866-5hkmc\" (UID: \"f65a88fe-acd7-4b42-808f-81f80c051a3e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-980661-68db9c5866-5hkmc"
Sep 18 14:18:48.844438 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:48.844368    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4hh7\" (UniqueName: \"kubernetes.io/projected/0e570098-8720-40dd-944b-28607af93d0a-kube-api-access-b4hh7\") pod \"payload-processing-e2e-shared-b-w3-980661-77765769cc-rvgpd\" (UID: \"0e570098-8720-40dd-944b-28607af93d0a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-980661-77765769cc-rvgpd"
Sep 18 14:18:48.844891 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:48.844872    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e570098-8720-40dd-944b-28607af93d0a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-980661-77765769cc-rvgpd\" (UID: \"0e570098-8720-40dd-944b-28607af93d0a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-980661-77765769cc-rvgpd"
Sep 18 14:18:48.854646 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:48.854620    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4hh7\" (UniqueName: \"kubernetes.io/projected/0e570098-8720-40dd-944b-28607af93d0a-kube-api-access-b4hh7\") pod \"payload-processing-e2e-shared-b-w3-980661-77765769cc-rvgpd\" (UID: \"0e570098-8720-40dd-944b-28607af93d0a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-980661-77765769cc-rvgpd"
Sep 18 14:18:48.945595 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:48.945562    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6rcc\" (UniqueName: \"kubernetes.io/projected/f65a88fe-acd7-4b42-808f-81f80c051a3e-kube-api-access-h6rcc\") pod \"payload-pre-processing-e2e-shared-b-w3-980661-68db9c5866-5hkmc\" (UID: \"f65a88fe-acd7-4b42-808f-81f80c051a3e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-980661-68db9c5866-5hkmc"
Sep 18 14:18:48.945719 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:48.945613    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f65a88fe-acd7-4b42-808f-81f80c051a3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-980661-68db9c5866-5hkmc\" (UID: \"f65a88fe-acd7-4b42-808f-81f80c051a3e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-980661-68db9c5866-5hkmc"
Sep 18 14:18:48.946364 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:48.946346    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f65a88fe-acd7-4b42-808f-81f80c051a3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-980661-68db9c5866-5hkmc\" (UID: \"f65a88fe-acd7-4b42-808f-81f80c051a3e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-980661-68db9c5866-5hkmc"
Sep 18 14:18:48.954901 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:48.954875    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6rcc\" (UniqueName: \"kubernetes.io/projected/f65a88fe-acd7-4b42-808f-81f80c051a3e-kube-api-access-h6rcc\") pod \"payload-pre-processing-e2e-shared-b-w3-980661-68db9c5866-5hkmc\" (UID: \"f65a88fe-acd7-4b42-808f-81f80c051a3e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-980661-68db9c5866-5hkmc"
Sep 18 14:18:49.022082 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:49.022059    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-980661-77765769cc-rvgpd"
Sep 18 14:18:49.062469 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:49.062425    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-980661-68db9c5866-5hkmc"
Sep 18 14:18:49.150628 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:49.150596    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:49.295693 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:49.295668    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-980661-77765769cc-rvgpd"]
Sep 18 14:18:49.297097 ip-10-0-143-120 kubenswrapper[2579]: W0918 14:18:49.297067    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e570098_8720_40dd_944b_28607af93d0a.slice/crio-db43d6fd5e46259678bde1281001fad7dc260c3cca3f269b6d79b7ad81aed77b WatchSource:0}: Error finding container db43d6fd5e46259678bde1281001fad7dc260c3cca3f269b6d79b7ad81aed77b: Status 404 returned error can't find the container with id db43d6fd5e46259678bde1281001fad7dc260c3cca3f269b6d79b7ad81aed77b
Sep 18 14:18:49.348683 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:49.348656    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-980661-68db9c5866-5hkmc"]
Sep 18 14:18:49.349579 ip-10-0-143-120 kubenswrapper[2579]: W0918 14:18:49.349559    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf65a88fe_acd7_4b42_808f_81f80c051a3e.slice/crio-9f6350296ba54e2b4c9fc67811188a14ab60f2ae379ab9c3e20620f1a1da0546 WatchSource:0}: Error finding container 9f6350296ba54e2b4c9fc67811188a14ab60f2ae379ab9c3e20620f1a1da0546: Status 404 returned error can't find the container with id 9f6350296ba54e2b4c9fc67811188a14ab60f2ae379ab9c3e20620f1a1da0546
Sep 18 14:18:49.380762 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:49.380720    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-980661-77765769cc-rvgpd" event={"ID":"0e570098-8720-40dd-944b-28607af93d0a","Type":"ContainerStarted","Data":"2ea5d5490e25d0fdec6e6e3a127cf52916389c1fa807b77d43d478bd36e025fa"}
Sep 18 14:18:49.380886 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:49.380790    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-980661-77765769cc-rvgpd" event={"ID":"0e570098-8720-40dd-944b-28607af93d0a","Type":"ContainerStarted","Data":"db43d6fd5e46259678bde1281001fad7dc260c3cca3f269b6d79b7ad81aed77b"}
Sep 18 14:18:49.380886 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:49.380821    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-980661-77765769cc-rvgpd"
Sep 18 14:18:49.382415 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:49.382394    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-980661-68db9c5866-5hkmc" event={"ID":"f65a88fe-acd7-4b42-808f-81f80c051a3e","Type":"ContainerStarted","Data":"9f6350296ba54e2b4c9fc67811188a14ab60f2ae379ab9c3e20620f1a1da0546"}
Sep 18 14:18:49.400849 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:49.400808    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-980661-77765769cc-rvgpd" podStartSLOduration=1.400794068 podStartE2EDuration="1.400794068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:18:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:18:49.399096275 +0000 UTC m=+1201.287812208" watchObservedRunningTime="2026-09-18 14:18:49.400794068 +0000 UTC m=+1201.289509998"
Sep 18 14:18:49.934711 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:49.934679    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:18:50.020458 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:50.020428    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sg4tv_4626a667-4cd6-468c-8ef3-6e3a925c2608/ovn-acl-logging/0.log"
Sep 18 14:18:50.021761 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:50.021737    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sg4tv_4626a667-4cd6-468c-8ef3-6e3a925c2608/ovn-acl-logging/0.log"
Sep 18 14:18:50.390157 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:50.390123    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-980661-68db9c5866-5hkmc" event={"ID":"f65a88fe-acd7-4b42-808f-81f80c051a3e","Type":"ContainerStarted","Data":"b4cf55ef37eb597f8bb447ecf0e1a4033ba57140f8909bd97590f11944612f45"}
Sep 18 14:18:50.390379 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:50.390356    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-980661-68db9c5866-5hkmc"
Sep 18 14:18:50.449154 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:18:50.449093    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-980661-68db9c5866-5hkmc" podStartSLOduration=2.449077886 podStartE2EDuration="2.449077886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:18:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:18:50.447658977 +0000 UTC m=+1202.336374908" watchObservedRunningTime="2026-09-18 14:18:50.449077886 +0000 UTC m=+1202.337793816"
Sep 18 14:19:00.392228 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:19:00.392191    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-980661-77765769cc-rvgpd"
Sep 18 14:19:01.397200 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:19:01.397145    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-980661-68db9c5866-5hkmc"
Sep 18 14:19:04.488195 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:19:04.487734    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:19:04.585888 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:19:04.585861    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:19:09.069189 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:19:09.069137    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:19:10.299133 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:19:10.299089    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:19:11.140363 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:19:11.140331    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:19:11.339798 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:19:11.339763    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Sep 18 14:19:11.340209 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:19:11.339828    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn" podUID="bec5d7ac-13a5-4317-b941-c8c47dbcb25e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Sep 18 14:19:11.519713 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:19:11.519623    2579 generic.go:356] "Generic (PLEG): container finished" podID="bec5d7ac-13a5-4317-b941-c8c47dbcb25e" containerID="0141d3ef082394e73b9a341213e8f15d3f9795aa96d23ee0952356a4f0bb04bd" exitCode=0
Sep 18 14:19:11.519713 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:19:11.519698    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn" event={"ID":"bec5d7ac-13a5-4317-b941-c8c47dbcb25e","Type":"ContainerDied","Data":"0141d3ef082394e73b9a341213e8f15d3f9795aa96d23ee0952356a4f0bb04bd"}
Sep 18 14:19:11.520161 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:19:11.520147    2579 scope.go:117] "RemoveContainer" containerID="0141d3ef082394e73b9a341213e8f15d3f9795aa96d23ee0952356a4f0bb04bd"
Sep 18 14:19:11.521396 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:19:11.521364    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:19:11.521491 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:19:11.521441    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:19:11.521491 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:19:11.521473    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:19:11.909617 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:19:11.909582    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:19:12.470757 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:19:12.470717    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:19:12.527569 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:19:12.527538    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn" event={"ID":"bec5d7ac-13a5-4317-b941-c8c47dbcb25e","Type":"ContainerStarted","Data":"ea7f312e95ff4116f8087b634bea4a3bdd6cd090a9e59b3ef85dd33295f32ebf"}
Sep 18 14:19:12.824085 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:19:12.824057    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:19:13.147071 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:19:13.146992    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn"
Sep 18 14:19:13.147239 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:19:13.147143    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Sep 18 14:19:13.147239 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:19:13.147217    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn" podUID="bec5d7ac-13a5-4317-b941-c8c47dbcb25e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Sep 18 14:19:13.736960 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:19:13.736919    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:19:14.147100 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:19:14.147069    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Sep 18 14:19:14.147298 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:19:14.147117    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn" podUID="bec5d7ac-13a5-4317-b941-c8c47dbcb25e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Sep 18 14:19:14.614017 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:19:14.613986    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:19:15.147685 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:19:15.147656    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Sep 18 14:19:15.148071 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:19:15.147703    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn" podUID="bec5d7ac-13a5-4317-b941-c8c47dbcb25e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Sep 18 14:19:15.349603 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:19:15.349574    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:19:16.147476 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:19:16.147442    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Sep 18 14:19:16.147651 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:19:16.147493    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn" podUID="bec5d7ac-13a5-4317-b941-c8c47dbcb25e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Sep 18 14:19:16.235677 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:19:16.235649    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:19:17.147061 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:19:17.147027    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Sep 18 14:19:17.147305 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:19:17.147080    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn" podUID="bec5d7ac-13a5-4317-b941-c8c47dbcb25e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Sep 18 14:19:18.152132 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:19:18.152104    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn"
Sep 18 14:19:18.152642 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:19:18.152623    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn"
Sep 18 14:19:18.153418 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:19:18.153393    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn"
Sep 18 14:19:19.010480 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:19:19.010451    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:20:02.589737 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:20:02.589655    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:20:37.662825 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:20:37.662793    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:20:47.518568 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:20:47.518530    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:20:48.666942 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:20:48.666899    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:20:49.674046 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:20:49.674005    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:20:49.968935 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:20:49.968855    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:20:50.078181 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:20:50.078134    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:20:50.764404 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:20:50.764350    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:20:51.874802 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:20:51.874772    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:20:52.761829 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:20:52.761795    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:20:53.559957 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:20:53.559915    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:20:54.667458 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:20:54.667427    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:20:55.469949 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:20:55.469916    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:21:00.536227 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:21:00.536191    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:21:00.637574 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:21:00.637538    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:21:00.712446 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:21:00.712417    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:21:02.940154 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:21:02.940081    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-988c9d38-6cc7984df-7hlvf"]
Sep 18 14:21:02.940540 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:21:02.940308    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-988c9d38-6cc7984df-7hlvf" podUID="26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b" containerName="maas-api" containerID="cri-o://5201ec87cc904bfcf469291b76db28dc67dca2245e10aab7b1b758201150101d" gracePeriod=30
Sep 18 14:21:03.021022 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:21:03.020992    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:21:03.083775 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:21:03.083341    2579 generic.go:356] "Generic (PLEG): container finished" podID="26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b" containerID="5201ec87cc904bfcf469291b76db28dc67dca2245e10aab7b1b758201150101d" exitCode=0
Sep 18 14:21:03.083775 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:21:03.083417    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-988c9d38-6cc7984df-7hlvf" event={"ID":"26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b","Type":"ContainerDied","Data":"5201ec87cc904bfcf469291b76db28dc67dca2245e10aab7b1b758201150101d"}
Sep 18 14:21:03.123601 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:21:03.123572    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:21:03.194811 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:21:03.194760    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-988c9d38-6cc7984df-7hlvf"
Sep 18 14:21:03.279412 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:21:03.279375    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b-openshift-service-ca\") pod \"26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b\" (UID: \"26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b\") "
Sep 18 14:21:03.279590 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:21:03.279445    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b-maas-api-tls\") pod \"26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b\" (UID: \"26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b\") "
Sep 18 14:21:03.279590 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:21:03.279487    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hd77\" (UniqueName: \"kubernetes.io/projected/26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b-kube-api-access-6hd77\") pod \"26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b\" (UID: \"26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b\") "
Sep 18 14:21:03.279590 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:21:03.279539    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b-metrics-certs\") pod \"26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b\" (UID: \"26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b\") "
Sep 18 14:21:03.279886 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:21:03.279859    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b" (UID: "26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:21:03.283528 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:21:03.282527    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b-kube-api-access-6hd77" (OuterVolumeSpecName: "kube-api-access-6hd77") pod "26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b" (UID: "26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b"). InnerVolumeSpecName "kube-api-access-6hd77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:21:03.283528 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:21:03.283297    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b" (UID: "26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:21:03.284284 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:21:03.284260    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b" (UID: "26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:21:03.310589 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:21:03.310554    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:21:03.380685 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:21:03.380650    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b-metrics-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:21:03.380685 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:21:03.380677    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b-openshift-service-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:21:03.380685 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:21:03.380687    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b-maas-api-tls\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:21:03.380966 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:21:03.380695    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hd77\" (UniqueName: \"kubernetes.io/projected/26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b-kube-api-access-6hd77\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:21:03.506858 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:21:03.506775    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:21:03.686547 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:21:03.686509    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:21:04.088953 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:21:04.088922    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-988c9d38-6cc7984df-7hlvf"
Sep 18 14:21:04.089380 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:21:04.088942    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-988c9d38-6cc7984df-7hlvf" event={"ID":"26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b","Type":"ContainerDied","Data":"c87cb570750e529f36275282431187494cf2bb1a3e268ac5f96cf5cf09dc2ad8"}
Sep 18 14:21:04.089380 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:21:04.089038    2579 scope.go:117] "RemoveContainer" containerID="5201ec87cc904bfcf469291b76db28dc67dca2245e10aab7b1b758201150101d"
Sep 18 14:21:04.123671 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:21:04.123650    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-988c9d38-6cc7984df-7hlvf"]
Sep 18 14:21:04.138292 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:21:04.138247    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-988c9d38-6cc7984df-7hlvf"]
Sep 18 14:21:04.227126 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:21:04.227096    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:21:05.257092 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:21:05.257058    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:21:06.035473 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:21:06.035441    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b" path="/var/lib/kubelet/pods/26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b/volumes"
Sep 18 14:21:06.248469 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:21:06.248412    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:21:07.056313 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:21:07.056277    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:21:07.546372 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:21:07.546344    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:21:09.283631 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:21:09.283591    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:21:10.347539 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:21:10.347507    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:21:11.352568 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:21:11.352537    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:21:12.102083 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:21:12.102052    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:21:18.814707 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:21:18.814669    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:21:34.255132 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:21:34.255097    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:21:39.651440 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:21:39.651402    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:21:49.066010 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:21:49.065981    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:21:49.521713 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:21:49.521641    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:21:50.494743 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:21:50.494668    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:21:51.038909 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:21:51.038878    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:21:51.139962 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:21:51.139929    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:21:56.168875 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:21:56.168839    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:22:02.438759 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:02.438723    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:22:02.536920 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:02.536881    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:22:02.737153 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:02.737061    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:22:04.068745 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:04.068710    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:22:04.166138 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:04.166100    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:22:04.873433 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:04.873398    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:22:04.987416 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:04.987384    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:22:06.065647 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:06.065613    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:22:06.180827 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:06.180795    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:22:06.434810 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:06.434734    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:22:12.795466 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:12.795431    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:22:12.895058 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:12.895029    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:22:13.480936 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:13.480904    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:22:19.673668 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:19.673636    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:22:19.748030 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:19.748003    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:22:19.841590 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:19.841564    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:22:30.411617 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:30.411572    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:22:30.634552 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:30.634481    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:22:40.889032 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:40.888957    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:22:41.368504 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:41.368469    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:22:42.115192 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:42.115132    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:22:42.838845 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:42.838812    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:22:43.651415 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:43.651344    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:22:44.333690 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:44.333663    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:22:45.161529 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:45.161500    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:22:45.865033 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:45.865000    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4d9222/distinct-w0-4d9222-kserve-77444cbdf6-bz5lg"]
Sep 18 14:22:45.956789 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:45.956750    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:22:46.058634 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:46.058600    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:22:46.670746 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:46.670709    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:22:47.556894 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:47.556859    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:22:51.140122 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:51.140084    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:22:51.576009 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:51.575973    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:22:56.849458 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:56.849422    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-4d9222-7d5bb89d47-c2shv"]
Sep 18 14:22:56.849915 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:56.849625    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4d9222-7d5bb89d47-c2shv" podUID="938f2892-9feb-4658-acc8-5303503bcf05" containerName="maas-api" containerID="cri-o://152f68837325a11e8fae70818ee11db240d3199b10a188116da4027fcf2dc893" gracePeriod=30
Sep 18 14:22:56.913430 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:56.913403    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:22:57.010593 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:57.010556    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:22:57.107146 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:57.107093    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4d9222-7d5bb89d47-c2shv"
Sep 18 14:22:57.114297 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:57.114272    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:22:57.218469 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:57.218437    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:22:57.260537 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:57.260501    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8m87\" (UniqueName: \"kubernetes.io/projected/938f2892-9feb-4658-acc8-5303503bcf05-kube-api-access-s8m87\") pod \"938f2892-9feb-4658-acc8-5303503bcf05\" (UID: \"938f2892-9feb-4658-acc8-5303503bcf05\") "
Sep 18 14:22:57.260739 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:57.260599    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/938f2892-9feb-4658-acc8-5303503bcf05-maas-api-tls\") pod \"938f2892-9feb-4658-acc8-5303503bcf05\" (UID: \"938f2892-9feb-4658-acc8-5303503bcf05\") "
Sep 18 14:22:57.260739 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:57.260624    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/938f2892-9feb-4658-acc8-5303503bcf05-metrics-certs\") pod \"938f2892-9feb-4658-acc8-5303503bcf05\" (UID: \"938f2892-9feb-4658-acc8-5303503bcf05\") "
Sep 18 14:22:57.260739 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:57.260657    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/938f2892-9feb-4658-acc8-5303503bcf05-openshift-service-ca\") pod \"938f2892-9feb-4658-acc8-5303503bcf05\" (UID: \"938f2892-9feb-4658-acc8-5303503bcf05\") "
Sep 18 14:22:57.261297 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:57.261266    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/938f2892-9feb-4658-acc8-5303503bcf05-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "938f2892-9feb-4658-acc8-5303503bcf05" (UID: "938f2892-9feb-4658-acc8-5303503bcf05"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:22:57.263969 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:57.263935    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/938f2892-9feb-4658-acc8-5303503bcf05-kube-api-access-s8m87" (OuterVolumeSpecName: "kube-api-access-s8m87") pod "938f2892-9feb-4658-acc8-5303503bcf05" (UID: "938f2892-9feb-4658-acc8-5303503bcf05"). InnerVolumeSpecName "kube-api-access-s8m87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:22:57.264080 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:57.264010    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/938f2892-9feb-4658-acc8-5303503bcf05-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "938f2892-9feb-4658-acc8-5303503bcf05" (UID: "938f2892-9feb-4658-acc8-5303503bcf05"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:22:57.264396 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:57.264373    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/938f2892-9feb-4658-acc8-5303503bcf05-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "938f2892-9feb-4658-acc8-5303503bcf05" (UID: "938f2892-9feb-4658-acc8-5303503bcf05"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:22:57.361390 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:57.361320    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/938f2892-9feb-4658-acc8-5303503bcf05-openshift-service-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:22:57.361390 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:57.361347    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8m87\" (UniqueName: \"kubernetes.io/projected/938f2892-9feb-4658-acc8-5303503bcf05-kube-api-access-s8m87\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:22:57.361390 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:57.361360    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/938f2892-9feb-4658-acc8-5303503bcf05-maas-api-tls\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:22:57.361390 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:57.361373    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/938f2892-9feb-4658-acc8-5303503bcf05-metrics-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:22:57.570453 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:57.570422    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:22:57.646883 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:57.646802    2579 generic.go:356] "Generic (PLEG): container finished" podID="938f2892-9feb-4658-acc8-5303503bcf05" containerID="152f68837325a11e8fae70818ee11db240d3199b10a188116da4027fcf2dc893" exitCode=0
Sep 18 14:22:57.646883 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:57.646849    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4d9222-7d5bb89d47-c2shv"
Sep 18 14:22:57.646883 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:57.646853    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4d9222-7d5bb89d47-c2shv" event={"ID":"938f2892-9feb-4658-acc8-5303503bcf05","Type":"ContainerDied","Data":"152f68837325a11e8fae70818ee11db240d3199b10a188116da4027fcf2dc893"}
Sep 18 14:22:57.647129 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:57.646927    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4d9222-7d5bb89d47-c2shv" event={"ID":"938f2892-9feb-4658-acc8-5303503bcf05","Type":"ContainerDied","Data":"ee439c8c240fe3e01d394afbd64a425006a9ccff02f2d4ed148551feb5283db0"}
Sep 18 14:22:57.647129 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:57.646960    2579 scope.go:117] "RemoveContainer" containerID="152f68837325a11e8fae70818ee11db240d3199b10a188116da4027fcf2dc893"
Sep 18 14:22:57.660047 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:57.660026    2579 scope.go:117] "RemoveContainer" containerID="152f68837325a11e8fae70818ee11db240d3199b10a188116da4027fcf2dc893"
Sep 18 14:22:57.660357 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:22:57.660333    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"152f68837325a11e8fae70818ee11db240d3199b10a188116da4027fcf2dc893\": container with ID starting with 152f68837325a11e8fae70818ee11db240d3199b10a188116da4027fcf2dc893 not found: ID does not exist" containerID="152f68837325a11e8fae70818ee11db240d3199b10a188116da4027fcf2dc893"
Sep 18 14:22:57.660421 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:57.660369    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"152f68837325a11e8fae70818ee11db240d3199b10a188116da4027fcf2dc893"} err="failed to get container status \"152f68837325a11e8fae70818ee11db240d3199b10a188116da4027fcf2dc893\": rpc error: code = NotFound desc = could not find container \"152f68837325a11e8fae70818ee11db240d3199b10a188116da4027fcf2dc893\": container with ID starting with 152f68837325a11e8fae70818ee11db240d3199b10a188116da4027fcf2dc893 not found: ID does not exist"
Sep 18 14:22:57.672435 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:57.672377    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:22:57.672906 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:57.672889    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-4d9222-7d5bb89d47-c2shv"]
Sep 18 14:22:57.674543 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:57.674525    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-4d9222-7d5bb89d47-c2shv"]
Sep 18 14:22:58.035996 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:58.035911    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="938f2892-9feb-4658-acc8-5303503bcf05" path="/var/lib/kubelet/pods/938f2892-9feb-4658-acc8-5303503bcf05/volumes"
Sep 18 14:22:59.070495 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:59.070464    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:22:59.330371 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:59.330290    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:22:59.430594 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:59.430564    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:22:59.632691 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:59.632615    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:22:59.956303 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:59.956219    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh"]
Sep 18 14:22:59.958636 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:59.958611    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b" containerName="maas-api"
Sep 18 14:22:59.958775 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:59.958671    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b" containerName="maas-api"
Sep 18 14:22:59.958775 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:59.958709    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="938f2892-9feb-4658-acc8-5303503bcf05" containerName="maas-api"
Sep 18 14:22:59.958775 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:59.958718    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="938f2892-9feb-4658-acc8-5303503bcf05" containerName="maas-api"
Sep 18 14:22:59.958966 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:59.958949    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="26dc2ae3-49a7-4fc1-b05e-f88d3dbe1d4b" containerName="maas-api"
Sep 18 14:22:59.959042 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:59.958973    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="938f2892-9feb-4658-acc8-5303503bcf05" containerName="maas-api"
Sep 18 14:22:59.964479 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:59.964458    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh"
Sep 18 14:22:59.967328 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:59.967303    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-e01050-openshift-default-dockercfg-lwtwm\""
Sep 18 14:22:59.990757 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:22:59.990731    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh"]
Sep 18 14:23:00.035589 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:00.035549    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:00.085964 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:00.085928    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c8c05d0-dc2f-494d-92a1-971574ddd421-istio-data\") pod \"e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh\" (UID: \"5c8c05d0-dc2f-494d-92a1-971574ddd421\") " pod="openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh"
Sep 18 14:23:00.086441 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:00.085978    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fckr\" (UniqueName: \"kubernetes.io/projected/5c8c05d0-dc2f-494d-92a1-971574ddd421-kube-api-access-8fckr\") pod \"e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh\" (UID: \"5c8c05d0-dc2f-494d-92a1-971574ddd421\") " pod="openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh"
Sep 18 14:23:00.086441 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:00.086001    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c8c05d0-dc2f-494d-92a1-971574ddd421-istio-podinfo\") pod \"e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh\" (UID: \"5c8c05d0-dc2f-494d-92a1-971574ddd421\") " pod="openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh"
Sep 18 14:23:00.086441 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:00.086081    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c8c05d0-dc2f-494d-92a1-971574ddd421-istio-token\") pod \"e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh\" (UID: \"5c8c05d0-dc2f-494d-92a1-971574ddd421\") " pod="openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh"
Sep 18 14:23:00.086441 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:00.086167    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c8c05d0-dc2f-494d-92a1-971574ddd421-workload-socket\") pod \"e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh\" (UID: \"5c8c05d0-dc2f-494d-92a1-971574ddd421\") " pod="openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh"
Sep 18 14:23:00.086441 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:00.086236    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c8c05d0-dc2f-494d-92a1-971574ddd421-istio-envoy\") pod \"e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh\" (UID: \"5c8c05d0-dc2f-494d-92a1-971574ddd421\") " pod="openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh"
Sep 18 14:23:00.086441 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:00.086304    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c8c05d0-dc2f-494d-92a1-971574ddd421-workload-certs\") pod \"e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh\" (UID: \"5c8c05d0-dc2f-494d-92a1-971574ddd421\") " pod="openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh"
Sep 18 14:23:00.086441 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:00.086331    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c8c05d0-dc2f-494d-92a1-971574ddd421-credential-socket\") pod \"e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh\" (UID: \"5c8c05d0-dc2f-494d-92a1-971574ddd421\") " pod="openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh"
Sep 18 14:23:00.086441 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:00.086362    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c8c05d0-dc2f-494d-92a1-971574ddd421-istiod-ca-cert\") pod \"e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh\" (UID: \"5c8c05d0-dc2f-494d-92a1-971574ddd421\") " pod="openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh"
Sep 18 14:23:00.187319 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:00.187294    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fckr\" (UniqueName: \"kubernetes.io/projected/5c8c05d0-dc2f-494d-92a1-971574ddd421-kube-api-access-8fckr\") pod \"e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh\" (UID: \"5c8c05d0-dc2f-494d-92a1-971574ddd421\") " pod="openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh"
Sep 18 14:23:00.187479 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:00.187334    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c8c05d0-dc2f-494d-92a1-971574ddd421-istio-podinfo\") pod \"e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh\" (UID: \"5c8c05d0-dc2f-494d-92a1-971574ddd421\") " pod="openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh"
Sep 18 14:23:00.187479 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:00.187363    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c8c05d0-dc2f-494d-92a1-971574ddd421-istio-token\") pod \"e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh\" (UID: \"5c8c05d0-dc2f-494d-92a1-971574ddd421\") " pod="openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh"
Sep 18 14:23:00.187479 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:00.187407    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c8c05d0-dc2f-494d-92a1-971574ddd421-workload-socket\") pod \"e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh\" (UID: \"5c8c05d0-dc2f-494d-92a1-971574ddd421\") " pod="openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh"
Sep 18 14:23:00.187479 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:00.187437    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c8c05d0-dc2f-494d-92a1-971574ddd421-istio-envoy\") pod \"e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh\" (UID: \"5c8c05d0-dc2f-494d-92a1-971574ddd421\") " pod="openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh"
Sep 18 14:23:00.187479 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:00.187475    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c8c05d0-dc2f-494d-92a1-971574ddd421-workload-certs\") pod \"e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh\" (UID: \"5c8c05d0-dc2f-494d-92a1-971574ddd421\") " pod="openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh"
Sep 18 14:23:00.187754 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:00.187511    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c8c05d0-dc2f-494d-92a1-971574ddd421-credential-socket\") pod \"e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh\" (UID: \"5c8c05d0-dc2f-494d-92a1-971574ddd421\") " pod="openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh"
Sep 18 14:23:00.187754 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:00.187535    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c8c05d0-dc2f-494d-92a1-971574ddd421-istiod-ca-cert\") pod \"e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh\" (UID: \"5c8c05d0-dc2f-494d-92a1-971574ddd421\") " pod="openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh"
Sep 18 14:23:00.187754 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:00.187616    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c8c05d0-dc2f-494d-92a1-971574ddd421-istio-data\") pod \"e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh\" (UID: \"5c8c05d0-dc2f-494d-92a1-971574ddd421\") " pod="openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh"
Sep 18 14:23:00.188019 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:00.187995    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c8c05d0-dc2f-494d-92a1-971574ddd421-credential-socket\") pod \"e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh\" (UID: \"5c8c05d0-dc2f-494d-92a1-971574ddd421\") " pod="openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh"
Sep 18 14:23:00.188216 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:00.188100    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c8c05d0-dc2f-494d-92a1-971574ddd421-workload-certs\") pod \"e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh\" (UID: \"5c8c05d0-dc2f-494d-92a1-971574ddd421\") " pod="openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh"
Sep 18 14:23:00.188400 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:00.188303    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c8c05d0-dc2f-494d-92a1-971574ddd421-istiod-ca-cert\") pod \"e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh\" (UID: \"5c8c05d0-dc2f-494d-92a1-971574ddd421\") " pod="openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh"
Sep 18 14:23:00.188400 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:00.188372    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c8c05d0-dc2f-494d-92a1-971574ddd421-workload-socket\") pod \"e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh\" (UID: \"5c8c05d0-dc2f-494d-92a1-971574ddd421\") " pod="openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh"
Sep 18 14:23:00.188589 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:00.188418    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c8c05d0-dc2f-494d-92a1-971574ddd421-istio-data\") pod \"e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh\" (UID: \"5c8c05d0-dc2f-494d-92a1-971574ddd421\") " pod="openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh"
Sep 18 14:23:00.188672 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:00.188608    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:00.189868 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:00.189846    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c8c05d0-dc2f-494d-92a1-971574ddd421-istio-envoy\") pod \"e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh\" (UID: \"5c8c05d0-dc2f-494d-92a1-971574ddd421\") " pod="openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh"
Sep 18 14:23:00.189978 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:00.189962    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c8c05d0-dc2f-494d-92a1-971574ddd421-istio-podinfo\") pod \"e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh\" (UID: \"5c8c05d0-dc2f-494d-92a1-971574ddd421\") " pod="openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh"
Sep 18 14:23:00.199053 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:00.199032    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c8c05d0-dc2f-494d-92a1-971574ddd421-istio-token\") pod \"e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh\" (UID: \"5c8c05d0-dc2f-494d-92a1-971574ddd421\") " pod="openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh"
Sep 18 14:23:00.199271 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:00.199254    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fckr\" (UniqueName: \"kubernetes.io/projected/5c8c05d0-dc2f-494d-92a1-971574ddd421-kube-api-access-8fckr\") pod \"e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh\" (UID: \"5c8c05d0-dc2f-494d-92a1-971574ddd421\") " pod="openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh"
Sep 18 14:23:00.275330 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:00.275249    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh"
Sep 18 14:23:00.284841 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:00.284812    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:00.556501 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:00.556462    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh"]
Sep 18 14:23:00.557094 ip-10-0-143-120 kubenswrapper[2579]: W0918 14:23:00.557053    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c8c05d0_dc2f_494d_92a1_971574ddd421.slice/crio-906803a771b8aee323d7cebdd1ea584b5690734ddd227071131f2a4aea3f18d6 WatchSource:0}: Error finding container 906803a771b8aee323d7cebdd1ea584b5690734ddd227071131f2a4aea3f18d6: Status 404 returned error can't find the container with id 906803a771b8aee323d7cebdd1ea584b5690734ddd227071131f2a4aea3f18d6
Sep 18 14:23:00.559758 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:00.559713    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:23:00.559845 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:00.559806    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:23:00.559885 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:00.559847    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:23:00.667182 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:00.667136    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh" event={"ID":"5c8c05d0-dc2f-494d-92a1-971574ddd421","Type":"ContainerStarted","Data":"4228653cddd5a7a7ef3c4a60c53cd22a331c4d55bf6aaf1fbe72183aa77208a8"}
Sep 18 14:23:00.667342 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:00.667210    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh" event={"ID":"5c8c05d0-dc2f-494d-92a1-971574ddd421","Type":"ContainerStarted","Data":"906803a771b8aee323d7cebdd1ea584b5690734ddd227071131f2a4aea3f18d6"}
Sep 18 14:23:00.689017 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:00.688973    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh" podStartSLOduration=1.688958608 podStartE2EDuration="1.688958608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:22:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:23:00.687783584 +0000 UTC m=+1452.576499517" watchObservedRunningTime="2026-09-18 14:23:00.688958608 +0000 UTC m=+1452.577674539"
Sep 18 14:23:00.886633 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:00.886545    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-4d9222/distinct-w0-4d9222-kserve-77444cbdf6-bz5lg" podUID="272988d2-cade-46e9-8fef-ad40e429f9b4" containerName="main" containerID="cri-o://ed4e69524692cec35a03e661facaac270865797c3ddbc31d9f01f9b7466ed607" gracePeriod=45
Sep 18 14:23:00.927218 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:00.927192    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:01.121822 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:01.121799    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4d9222/distinct-w0-4d9222-kserve-77444cbdf6-bz5lg"
Sep 18 14:23:01.275913 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:01.275883    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh"
Sep 18 14:23:01.281655 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:01.281630    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh"
Sep 18 14:23:01.296006 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:01.295969    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/272988d2-cade-46e9-8fef-ad40e429f9b4-home\") pod \"272988d2-cade-46e9-8fef-ad40e429f9b4\" (UID: \"272988d2-cade-46e9-8fef-ad40e429f9b4\") "
Sep 18 14:23:01.296143 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:01.296025    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/272988d2-cade-46e9-8fef-ad40e429f9b4-tmp-dir\") pod \"272988d2-cade-46e9-8fef-ad40e429f9b4\" (UID: \"272988d2-cade-46e9-8fef-ad40e429f9b4\") "
Sep 18 14:23:01.296143 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:01.296065    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/272988d2-cade-46e9-8fef-ad40e429f9b4-dshm\") pod \"272988d2-cade-46e9-8fef-ad40e429f9b4\" (UID: \"272988d2-cade-46e9-8fef-ad40e429f9b4\") "
Sep 18 14:23:01.296143 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:01.296094    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/272988d2-cade-46e9-8fef-ad40e429f9b4-model-cache\") pod \"272988d2-cade-46e9-8fef-ad40e429f9b4\" (UID: \"272988d2-cade-46e9-8fef-ad40e429f9b4\") "
Sep 18 14:23:01.296143 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:01.296120    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/272988d2-cade-46e9-8fef-ad40e429f9b4-tls-certs\") pod \"272988d2-cade-46e9-8fef-ad40e429f9b4\" (UID: \"272988d2-cade-46e9-8fef-ad40e429f9b4\") "
Sep 18 14:23:01.296410 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:01.296203    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bfffg\" (UniqueName: \"kubernetes.io/projected/272988d2-cade-46e9-8fef-ad40e429f9b4-kube-api-access-bfffg\") pod \"272988d2-cade-46e9-8fef-ad40e429f9b4\" (UID: \"272988d2-cade-46e9-8fef-ad40e429f9b4\") "
Sep 18 14:23:01.296410 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:01.296362    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/272988d2-cade-46e9-8fef-ad40e429f9b4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "272988d2-cade-46e9-8fef-ad40e429f9b4" (UID: "272988d2-cade-46e9-8fef-ad40e429f9b4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:23:01.296531 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:01.296477    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/272988d2-cade-46e9-8fef-ad40e429f9b4-model-cache" (OuterVolumeSpecName: "model-cache") pod "272988d2-cade-46e9-8fef-ad40e429f9b4" (UID: "272988d2-cade-46e9-8fef-ad40e429f9b4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:23:01.296620 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:01.296602    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/272988d2-cade-46e9-8fef-ad40e429f9b4-tmp-dir\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:23:01.296681 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:01.296626    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/272988d2-cade-46e9-8fef-ad40e429f9b4-model-cache\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:23:01.296740 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:01.296676    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/272988d2-cade-46e9-8fef-ad40e429f9b4-home" (OuterVolumeSpecName: "home") pod "272988d2-cade-46e9-8fef-ad40e429f9b4" (UID: "272988d2-cade-46e9-8fef-ad40e429f9b4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:23:01.298860 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:01.298829    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/272988d2-cade-46e9-8fef-ad40e429f9b4-kube-api-access-bfffg" (OuterVolumeSpecName: "kube-api-access-bfffg") pod "272988d2-cade-46e9-8fef-ad40e429f9b4" (UID: "272988d2-cade-46e9-8fef-ad40e429f9b4"). InnerVolumeSpecName "kube-api-access-bfffg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:23:01.298987 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:01.298887    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/272988d2-cade-46e9-8fef-ad40e429f9b4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "272988d2-cade-46e9-8fef-ad40e429f9b4" (UID: "272988d2-cade-46e9-8fef-ad40e429f9b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:23:01.299052 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:01.298995    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/272988d2-cade-46e9-8fef-ad40e429f9b4-dshm" (OuterVolumeSpecName: "dshm") pod "272988d2-cade-46e9-8fef-ad40e429f9b4" (UID: "272988d2-cade-46e9-8fef-ad40e429f9b4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:23:01.397703 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:01.397675    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/272988d2-cade-46e9-8fef-ad40e429f9b4-home\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:23:01.397703 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:01.397700    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/272988d2-cade-46e9-8fef-ad40e429f9b4-dshm\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:23:01.397703 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:01.397710    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/272988d2-cade-46e9-8fef-ad40e429f9b4-tls-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:23:01.397918 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:01.397720    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfffg\" (UniqueName: \"kubernetes.io/projected/272988d2-cade-46e9-8fef-ad40e429f9b4-kube-api-access-bfffg\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:23:01.673143 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:01.673109    2579 generic.go:356] "Generic (PLEG): container finished" podID="272988d2-cade-46e9-8fef-ad40e429f9b4" containerID="ed4e69524692cec35a03e661facaac270865797c3ddbc31d9f01f9b7466ed607" exitCode=0
Sep 18 14:23:01.673322 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:01.673154    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4d9222/distinct-w0-4d9222-kserve-77444cbdf6-bz5lg"
Sep 18 14:23:01.673322 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:01.673195    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4d9222/distinct-w0-4d9222-kserve-77444cbdf6-bz5lg" event={"ID":"272988d2-cade-46e9-8fef-ad40e429f9b4","Type":"ContainerDied","Data":"ed4e69524692cec35a03e661facaac270865797c3ddbc31d9f01f9b7466ed607"}
Sep 18 14:23:01.673322 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:01.673234    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4d9222/distinct-w0-4d9222-kserve-77444cbdf6-bz5lg" event={"ID":"272988d2-cade-46e9-8fef-ad40e429f9b4","Type":"ContainerDied","Data":"49f7f42df7987e6874a859b6bba394097d7578268d865673aad11134846081a6"}
Sep 18 14:23:01.673322 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:01.673257    2579 scope.go:117] "RemoveContainer" containerID="ed4e69524692cec35a03e661facaac270865797c3ddbc31d9f01f9b7466ed607"
Sep 18 14:23:01.673616 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:01.673598    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh"
Sep 18 14:23:01.674800 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:01.674779    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh"
Sep 18 14:23:01.682939 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:01.682924    2579 scope.go:117] "RemoveContainer" containerID="ed4e69524692cec35a03e661facaac270865797c3ddbc31d9f01f9b7466ed607"
Sep 18 14:23:01.683222 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:23:01.683203    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed4e69524692cec35a03e661facaac270865797c3ddbc31d9f01f9b7466ed607\": container with ID starting with ed4e69524692cec35a03e661facaac270865797c3ddbc31d9f01f9b7466ed607 not found: ID does not exist" containerID="ed4e69524692cec35a03e661facaac270865797c3ddbc31d9f01f9b7466ed607"
Sep 18 14:23:01.683281 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:01.683230    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed4e69524692cec35a03e661facaac270865797c3ddbc31d9f01f9b7466ed607"} err="failed to get container status \"ed4e69524692cec35a03e661facaac270865797c3ddbc31d9f01f9b7466ed607\": rpc error: code = NotFound desc = could not find container \"ed4e69524692cec35a03e661facaac270865797c3ddbc31d9f01f9b7466ed607\": container with ID starting with ed4e69524692cec35a03e661facaac270865797c3ddbc31d9f01f9b7466ed607 not found: ID does not exist"
Sep 18 14:23:01.712930 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:01.712860    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:01.714212 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:01.714189    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4d9222/distinct-w0-4d9222-kserve-77444cbdf6-bz5lg"]
Sep 18 14:23:01.727296 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:01.727272    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-4d9222/distinct-w0-4d9222-kserve-77444cbdf6-bz5lg"]
Sep 18 14:23:02.035337 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:02.035242    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="272988d2-cade-46e9-8fef-ad40e429f9b4" path="/var/lib/kubelet/pods/272988d2-cade-46e9-8fef-ad40e429f9b4/volumes"
Sep 18 14:23:02.640524 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:02.640461    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:02.739335 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:02.739297    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:03.527107 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:03.527075    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:04.332472 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:04.332440    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:04.433303 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:04.433266    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:04.535379 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:04.535345    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:04.677827 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:04.677722    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:05.195095 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:05.195058    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:06.138189 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:06.138139    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:07.178115 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:07.178084    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:08.142642 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:08.142594    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:08.211513 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:08.211479    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:08.312203 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:08.312125    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:08.623306 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:08.623264    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:08.698279 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:08.698241    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:08.738357 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:08.738316    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e01050-69565475c8-fdh98"]
Sep 18 14:23:08.738782 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:08.738763    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="272988d2-cade-46e9-8fef-ad40e429f9b4" containerName="main"
Sep 18 14:23:08.738884 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:08.738784    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="272988d2-cade-46e9-8fef-ad40e429f9b4" containerName="main"
Sep 18 14:23:08.738939 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:08.738906    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="272988d2-cade-46e9-8fef-ad40e429f9b4" containerName="main"
Sep 18 14:23:08.743296 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:08.743272    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e01050-69565475c8-fdh98"
Sep 18 14:23:08.746599 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:08.746576    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-e01050\""
Sep 18 14:23:08.747682 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:08.747666    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-e01050-dockercfg-dwgkc\""
Sep 18 14:23:08.753795 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:08.753767    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e01050-69565475c8-fdh98"]
Sep 18 14:23:08.864047 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:08.864002    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh56t\" (UniqueName: \"kubernetes.io/projected/26c2ff1c-e079-40da-933c-44a76190b5c1-kube-api-access-rh56t\") pod \"payload-pre-processing-e2e-api-a-e01050-69565475c8-fdh98\" (UID: \"26c2ff1c-e079-40da-933c-44a76190b5c1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e01050-69565475c8-fdh98"
Sep 18 14:23:08.864270 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:08.864143    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26c2ff1c-e079-40da-933c-44a76190b5c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e01050-69565475c8-fdh98\" (UID: \"26c2ff1c-e079-40da-933c-44a76190b5c1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e01050-69565475c8-fdh98"
Sep 18 14:23:08.965548 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:08.965451    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26c2ff1c-e079-40da-933c-44a76190b5c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e01050-69565475c8-fdh98\" (UID: \"26c2ff1c-e079-40da-933c-44a76190b5c1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e01050-69565475c8-fdh98"
Sep 18 14:23:08.965548 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:08.965545    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh56t\" (UniqueName: \"kubernetes.io/projected/26c2ff1c-e079-40da-933c-44a76190b5c1-kube-api-access-rh56t\") pod \"payload-pre-processing-e2e-api-a-e01050-69565475c8-fdh98\" (UID: \"26c2ff1c-e079-40da-933c-44a76190b5c1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e01050-69565475c8-fdh98"
Sep 18 14:23:08.966218 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:08.966198    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26c2ff1c-e079-40da-933c-44a76190b5c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e01050-69565475c8-fdh98\" (UID: \"26c2ff1c-e079-40da-933c-44a76190b5c1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e01050-69565475c8-fdh98"
Sep 18 14:23:08.974379 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:08.974352    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh56t\" (UniqueName: \"kubernetes.io/projected/26c2ff1c-e079-40da-933c-44a76190b5c1-kube-api-access-rh56t\") pod \"payload-pre-processing-e2e-api-a-e01050-69565475c8-fdh98\" (UID: \"26c2ff1c-e079-40da-933c-44a76190b5c1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e01050-69565475c8-fdh98"
Sep 18 14:23:09.048210 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:09.048156    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:09.055253 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:09.055224    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e01050-69565475c8-fdh98"
Sep 18 14:23:09.339501 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:09.339478    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e01050-69565475c8-fdh98"]
Sep 18 14:23:09.341671 ip-10-0-143-120 kubenswrapper[2579]: W0918 14:23:09.341643    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26c2ff1c_e079_40da_933c_44a76190b5c1.slice/crio-9f368372ef51dd0c65ee1740709b604603825083ed7c5888422e037d1c34cc17 WatchSource:0}: Error finding container 9f368372ef51dd0c65ee1740709b604603825083ed7c5888422e037d1c34cc17: Status 404 returned error can't find the container with id 9f368372ef51dd0c65ee1740709b604603825083ed7c5888422e037d1c34cc17
Sep 18 14:23:09.716355 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:09.716273    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e01050-69565475c8-fdh98" event={"ID":"26c2ff1c-e079-40da-933c-44a76190b5c1","Type":"ContainerStarted","Data":"eb58f2d71512b02bd2d2086af6615a0f1312357d24ad07bfb28324227fd78b81"}
Sep 18 14:23:09.716355 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:09.716338    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e01050-69565475c8-fdh98"
Sep 18 14:23:09.716540 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:09.716364    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e01050-69565475c8-fdh98" event={"ID":"26c2ff1c-e079-40da-933c-44a76190b5c1","Type":"ContainerStarted","Data":"9f368372ef51dd0c65ee1740709b604603825083ed7c5888422e037d1c34cc17"}
Sep 18 14:23:09.734727 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:09.734678    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e01050-69565475c8-fdh98" podStartSLOduration=1.7346635940000001 podStartE2EDuration="1.734663594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:23:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:23:09.732385429 +0000 UTC m=+1461.621101359" watchObservedRunningTime="2026-09-18 14:23:09.734663594 +0000 UTC m=+1461.623379570"
Sep 18 14:23:09.824328 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:09.824292    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:13.241937 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:13.241899    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:20.723968 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:20.723915    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e01050-69565475c8-fdh98"
Sep 18 14:23:21.276116 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:21.276081    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:21.375926 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:21.375898    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:21.813389 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:21.813357    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:22.761060 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:22.761028    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:23.425595 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:23.425563    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:23.526251 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:23.526220    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:23.626610 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:23.626573    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:24.067223 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:24.067186    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:24.254151 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:24.254116    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:25.053777 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:25.053751    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:25.471480 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:25.471407    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:25.796876 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:25.796834    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:26.628218 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:26.628180    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:27.324275 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:27.324244    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:27.583632 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:27.583602    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:28.084140 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:28.084110    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:28.489321 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:28.489232    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:28.579356 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:28.579327    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:28.676988 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:28.676952    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:29.311536 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:29.311507    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:30.227677 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:30.227646    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:30.897642 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:30.897608    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:31.334711 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:31.334676    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:31.434057 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:31.434019    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:32.394254 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:32.394224    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:33.550405 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:33.550374    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:34.291472 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:34.291425    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:34.390975 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:34.390941    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:34.544036 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:34.543919    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:34.753625 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:34.753559    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:34.858492 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:34.858436    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:35.152877 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:35.152802    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:35.968948 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:35.968908    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:36.642756 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:36.642727    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:37.689588 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:37.689547    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:38.426512 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:38.426477    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:39.342388 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:39.342342    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:39.921626 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:39.921581    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-93827d-cdb46b949-4rwcm"]
Sep 18 14:23:39.921905 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:39.921838    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-93827d-cdb46b949-4rwcm" podUID="0d516717-4809-4020-af5b-3cb8478f112d" containerName="maas-api" containerID="cri-o://013ce6a9dcbce1c2636301d63fcdc205e28a74037a2f59255af46dbba9de9bff" gracePeriod=30
Sep 18 14:23:39.951524 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:39.951499    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:40.038533 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:40.038501    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:40.101795 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:40.101707    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:40.192537 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:40.192510    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-93827d-cdb46b949-4rwcm"
Sep 18 14:23:40.204046 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:40.204029    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:40.363659 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:40.363609    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d516717-4809-4020-af5b-3cb8478f112d-maas-api-tls\") pod \"0d516717-4809-4020-af5b-3cb8478f112d\" (UID: \"0d516717-4809-4020-af5b-3cb8478f112d\") "
Sep 18 14:23:40.367668 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:40.364536    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d516717-4809-4020-af5b-3cb8478f112d-openshift-service-ca\") pod \"0d516717-4809-4020-af5b-3cb8478f112d\" (UID: \"0d516717-4809-4020-af5b-3cb8478f112d\") "
Sep 18 14:23:40.367668 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:40.364580    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d516717-4809-4020-af5b-3cb8478f112d-metrics-certs\") pod \"0d516717-4809-4020-af5b-3cb8478f112d\" (UID: \"0d516717-4809-4020-af5b-3cb8478f112d\") "
Sep 18 14:23:40.367668 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:40.364631    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ms6sl\" (UniqueName: \"kubernetes.io/projected/0d516717-4809-4020-af5b-3cb8478f112d-kube-api-access-ms6sl\") pod \"0d516717-4809-4020-af5b-3cb8478f112d\" (UID: \"0d516717-4809-4020-af5b-3cb8478f112d\") "
Sep 18 14:23:40.367668 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:40.365156    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d516717-4809-4020-af5b-3cb8478f112d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0d516717-4809-4020-af5b-3cb8478f112d" (UID: "0d516717-4809-4020-af5b-3cb8478f112d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:23:40.367668 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:40.367601    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d516717-4809-4020-af5b-3cb8478f112d-openshift-service-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:23:40.375594 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:40.374289    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d516717-4809-4020-af5b-3cb8478f112d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0d516717-4809-4020-af5b-3cb8478f112d" (UID: "0d516717-4809-4020-af5b-3cb8478f112d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:23:40.375594 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:40.374391    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d516717-4809-4020-af5b-3cb8478f112d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0d516717-4809-4020-af5b-3cb8478f112d" (UID: "0d516717-4809-4020-af5b-3cb8478f112d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:23:40.376870 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:40.376844    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d516717-4809-4020-af5b-3cb8478f112d-kube-api-access-ms6sl" (OuterVolumeSpecName: "kube-api-access-ms6sl") pod "0d516717-4809-4020-af5b-3cb8478f112d" (UID: "0d516717-4809-4020-af5b-3cb8478f112d"). InnerVolumeSpecName "kube-api-access-ms6sl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:23:40.468354 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:40.468289    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d516717-4809-4020-af5b-3cb8478f112d-maas-api-tls\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:23:40.468354 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:40.468309    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d516717-4809-4020-af5b-3cb8478f112d-metrics-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:23:40.468354 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:40.468320    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms6sl\" (UniqueName: \"kubernetes.io/projected/0d516717-4809-4020-af5b-3cb8478f112d-kube-api-access-ms6sl\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:23:40.500541 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:40.500516    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:40.634605 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:40.634580    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:40.875419 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:40.875382    2579 generic.go:356] "Generic (PLEG): container finished" podID="0d516717-4809-4020-af5b-3cb8478f112d" containerID="013ce6a9dcbce1c2636301d63fcdc205e28a74037a2f59255af46dbba9de9bff" exitCode=0
Sep 18 14:23:40.875577 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:40.875430    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-93827d-cdb46b949-4rwcm"
Sep 18 14:23:40.875577 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:40.875436    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-93827d-cdb46b949-4rwcm" event={"ID":"0d516717-4809-4020-af5b-3cb8478f112d","Type":"ContainerDied","Data":"013ce6a9dcbce1c2636301d63fcdc205e28a74037a2f59255af46dbba9de9bff"}
Sep 18 14:23:40.875577 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:40.875503    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-93827d-cdb46b949-4rwcm" event={"ID":"0d516717-4809-4020-af5b-3cb8478f112d","Type":"ContainerDied","Data":"a0b0bca4643d6697800d55556dbadff60b593c20229b9dacf922764be8a230dd"}
Sep 18 14:23:40.875577 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:40.875525    2579 scope.go:117] "RemoveContainer" containerID="013ce6a9dcbce1c2636301d63fcdc205e28a74037a2f59255af46dbba9de9bff"
Sep 18 14:23:40.887805 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:40.887784    2579 scope.go:117] "RemoveContainer" containerID="013ce6a9dcbce1c2636301d63fcdc205e28a74037a2f59255af46dbba9de9bff"
Sep 18 14:23:40.888221 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:23:40.888197    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"013ce6a9dcbce1c2636301d63fcdc205e28a74037a2f59255af46dbba9de9bff\": container with ID starting with 013ce6a9dcbce1c2636301d63fcdc205e28a74037a2f59255af46dbba9de9bff not found: ID does not exist" containerID="013ce6a9dcbce1c2636301d63fcdc205e28a74037a2f59255af46dbba9de9bff"
Sep 18 14:23:40.888287 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:40.888230    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"013ce6a9dcbce1c2636301d63fcdc205e28a74037a2f59255af46dbba9de9bff"} err="failed to get container status \"013ce6a9dcbce1c2636301d63fcdc205e28a74037a2f59255af46dbba9de9bff\": rpc error: code = NotFound desc = could not find container \"013ce6a9dcbce1c2636301d63fcdc205e28a74037a2f59255af46dbba9de9bff\": container with ID starting with 013ce6a9dcbce1c2636301d63fcdc205e28a74037a2f59255af46dbba9de9bff not found: ID does not exist"
Sep 18 14:23:40.899796 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:40.899761    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-93827d-cdb46b949-4rwcm"]
Sep 18 14:23:40.903403 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:40.903376    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-93827d-cdb46b949-4rwcm"]
Sep 18 14:23:41.333059 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:41.333008    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:41.883535 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:41.883501    2579 generic.go:356] "Generic (PLEG): container finished" podID="5c8c05d0-dc2f-494d-92a1-971574ddd421" containerID="4228653cddd5a7a7ef3c4a60c53cd22a331c4d55bf6aaf1fbe72183aa77208a8" exitCode=0
Sep 18 14:23:41.883905 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:41.883572    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh" event={"ID":"5c8c05d0-dc2f-494d-92a1-971574ddd421","Type":"ContainerDied","Data":"4228653cddd5a7a7ef3c4a60c53cd22a331c4d55bf6aaf1fbe72183aa77208a8"}
Sep 18 14:23:41.883950 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:41.883937    2579 scope.go:117] "RemoveContainer" containerID="4228653cddd5a7a7ef3c4a60c53cd22a331c4d55bf6aaf1fbe72183aa77208a8"
Sep 18 14:23:41.885036 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:41.885010    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:23:41.885094 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:41.885078    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:23:41.885133 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:41.885113    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:23:42.036969 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:42.036936    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d516717-4809-4020-af5b-3cb8478f112d" path="/var/lib/kubelet/pods/0d516717-4809-4020-af5b-3cb8478f112d/volumes"
Sep 18 14:23:42.234426 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:42.234353    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:42.891473 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:42.891439    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh" event={"ID":"5c8c05d0-dc2f-494d-92a1-971574ddd421","Type":"ContainerStarted","Data":"d0da1fd7051779ba08032032d6889b0ce3fdbb1a54e240dc84166af2759ec276"}
Sep 18 14:23:42.926865 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:42.926839    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:43.276303 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:43.276219    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh"
Sep 18 14:23:43.277400 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:43.277377    2579 patch_prober.go:28] interesting pod/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 18 14:23:43.277573 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:43.277550    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh" podUID="5c8c05d0-dc2f-494d-92a1-971574ddd421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 18 14:23:43.607584 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:43.607550    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:44.276527 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:44.276493    2579 patch_prober.go:28] interesting pod/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 18 14:23:44.277102 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:44.277062    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh" podUID="5c8c05d0-dc2f-494d-92a1-971574ddd421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 18 14:23:44.635386 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:44.635351    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:45.276088 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:45.276040    2579 patch_prober.go:28] interesting pod/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 18 14:23:45.276325 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:45.276143    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh" podUID="5c8c05d0-dc2f-494d-92a1-971574ddd421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 18 14:23:45.351865 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:45.351827    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:45.996198 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:45.996153    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:46.143418 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:46.143383    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:46.276348 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:46.276247    2579 patch_prober.go:28] interesting pod/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 18 14:23:46.276527 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:46.276356    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh" podUID="5c8c05d0-dc2f-494d-92a1-971574ddd421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 18 14:23:46.674600 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:46.674565    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh"
Sep 18 14:23:46.944829 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:46.944732    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:47.276385 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:47.276294    2579 patch_prober.go:28] interesting pod/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 18 14:23:47.276673 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:47.276620    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh" podUID="5c8c05d0-dc2f-494d-92a1-971574ddd421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 18 14:23:47.750634 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:47.750594    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:47.850826 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:47.850795    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:48.028257 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:48.028156    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:48.280409 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:48.280327    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh"
Sep 18 14:23:48.281293 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:48.281274    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh"
Sep 18 14:23:48.741818 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:48.741788    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:49.491112 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:49.491079    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:50.059153 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:50.059124    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sg4tv_4626a667-4cd6-468c-8ef3-6e3a925c2608/ovn-acl-logging/0.log"
Sep 18 14:23:50.062658 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:50.062636    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sg4tv_4626a667-4cd6-468c-8ef3-6e3a925c2608/ovn-acl-logging/0.log"
Sep 18 14:23:50.272185 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:50.272150    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:51.129001 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:51.128960    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:51.837087 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:51.837056    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:52.629665 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:52.629636    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:53.336984 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:53.336950    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:54.024617 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:54.024568    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:55.223619 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:55.223583    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:23:58.869544 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:23:58.869507    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:01.731994 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:01.731941    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:01.831805 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:01.831774    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:03.218661 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:03.218582    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:04.458077 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:04.458038    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:04.493739 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:04.493709    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e01050-69565475c8-fdh98"]
Sep 18 14:24:04.494009 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:04.493979    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e01050-69565475c8-fdh98" podUID="26c2ff1c-e079-40da-933c-44a76190b5c1" containerName="payload-pre-processing" containerID="cri-o://eb58f2d71512b02bd2d2086af6615a0f1312357d24ad07bfb28324227fd78b81" gracePeriod=30
Sep 18 14:24:04.557183 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:04.557144    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:04.659877 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:04.659841    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:04.740936 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:04.740761    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:04.752993 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:04.752828    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e01050-69565475c8-fdh98"
Sep 18 14:24:04.779867 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:04.779832    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rh56t\" (UniqueName: \"kubernetes.io/projected/26c2ff1c-e079-40da-933c-44a76190b5c1-kube-api-access-rh56t\") pod \"26c2ff1c-e079-40da-933c-44a76190b5c1\" (UID: \"26c2ff1c-e079-40da-933c-44a76190b5c1\") "
Sep 18 14:24:04.780080 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:04.779879    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26c2ff1c-e079-40da-933c-44a76190b5c1-plugins-config-volume\") pod \"26c2ff1c-e079-40da-933c-44a76190b5c1\" (UID: \"26c2ff1c-e079-40da-933c-44a76190b5c1\") "
Sep 18 14:24:04.780390 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:04.780357    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26c2ff1c-e079-40da-933c-44a76190b5c1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "26c2ff1c-e079-40da-933c-44a76190b5c1" (UID: "26c2ff1c-e079-40da-933c-44a76190b5c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:24:04.782036 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:04.782011    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26c2ff1c-e079-40da-933c-44a76190b5c1-kube-api-access-rh56t" (OuterVolumeSpecName: "kube-api-access-rh56t") pod "26c2ff1c-e079-40da-933c-44a76190b5c1" (UID: "26c2ff1c-e079-40da-933c-44a76190b5c1"). InnerVolumeSpecName "kube-api-access-rh56t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:24:04.881230 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:04.881135    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh56t\" (UniqueName: \"kubernetes.io/projected/26c2ff1c-e079-40da-933c-44a76190b5c1-kube-api-access-rh56t\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:24:04.881230 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:04.881191    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26c2ff1c-e079-40da-933c-44a76190b5c1-plugins-config-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:24:05.000955 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:05.000866    2579 generic.go:356] "Generic (PLEG): container finished" podID="26c2ff1c-e079-40da-933c-44a76190b5c1" containerID="eb58f2d71512b02bd2d2086af6615a0f1312357d24ad07bfb28324227fd78b81" exitCode=0
Sep 18 14:24:05.000955 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:05.000916    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e01050-69565475c8-fdh98"
Sep 18 14:24:05.000955 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:05.000919    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e01050-69565475c8-fdh98" event={"ID":"26c2ff1c-e079-40da-933c-44a76190b5c1","Type":"ContainerDied","Data":"eb58f2d71512b02bd2d2086af6615a0f1312357d24ad07bfb28324227fd78b81"}
Sep 18 14:24:05.001228 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:05.000970    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e01050-69565475c8-fdh98" event={"ID":"26c2ff1c-e079-40da-933c-44a76190b5c1","Type":"ContainerDied","Data":"9f368372ef51dd0c65ee1740709b604603825083ed7c5888422e037d1c34cc17"}
Sep 18 14:24:05.001228 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:05.001000    2579 scope.go:117] "RemoveContainer" containerID="eb58f2d71512b02bd2d2086af6615a0f1312357d24ad07bfb28324227fd78b81"
Sep 18 14:24:05.008653 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:05.008612    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:05.010438 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:05.010420    2579 scope.go:117] "RemoveContainer" containerID="eb58f2d71512b02bd2d2086af6615a0f1312357d24ad07bfb28324227fd78b81"
Sep 18 14:24:05.010715 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:24:05.010697    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb58f2d71512b02bd2d2086af6615a0f1312357d24ad07bfb28324227fd78b81\": container with ID starting with eb58f2d71512b02bd2d2086af6615a0f1312357d24ad07bfb28324227fd78b81 not found: ID does not exist" containerID="eb58f2d71512b02bd2d2086af6615a0f1312357d24ad07bfb28324227fd78b81"
Sep 18 14:24:05.010778 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:05.010727    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb58f2d71512b02bd2d2086af6615a0f1312357d24ad07bfb28324227fd78b81"} err="failed to get container status \"eb58f2d71512b02bd2d2086af6615a0f1312357d24ad07bfb28324227fd78b81\": rpc error: code = NotFound desc = could not find container \"eb58f2d71512b02bd2d2086af6615a0f1312357d24ad07bfb28324227fd78b81\": container with ID starting with eb58f2d71512b02bd2d2086af6615a0f1312357d24ad07bfb28324227fd78b81 not found: ID does not exist"
Sep 18 14:24:05.021824 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:05.021791    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e01050-69565475c8-fdh98"]
Sep 18 14:24:05.026294 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:05.026263    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e01050-69565475c8-fdh98"]
Sep 18 14:24:05.202765 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:05.202729    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:06.031629 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:06.031591    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:06.035189 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:06.035152    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26c2ff1c-e079-40da-933c-44a76190b5c1" path="/var/lib/kubelet/pods/26c2ff1c-e079-40da-933c-44a76190b5c1/volumes"
Sep 18 14:24:06.733452 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:06.733418    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:07.456611 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:07.456584    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:07.556974 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:07.556949    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:07.658133 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:07.658102    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:08.290091 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:08.290063    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:09.082738 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:09.082709    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:09.696929 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:09.696896    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:10.770627 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:10.770598    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:10.837196 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:10.837156    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:11.000343 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:11.000309    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh"]
Sep 18 14:24:11.000594 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:11.000568    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh" podUID="5c8c05d0-dc2f-494d-92a1-971574ddd421" containerName="istio-proxy" containerID="cri-o://d0da1fd7051779ba08032032d6889b0ce3fdbb1a54e240dc84166af2759ec276" gracePeriod=30
Sep 18 14:24:11.018270 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:11.018246    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:11.623162 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:11.623131    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:12.428521 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:12.428484    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:13.109132 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:13.109097    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:13.826460 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:13.826425    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:15.078789 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:15.078751    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:15.762991 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:15.762924    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:16.256069 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:16.256045    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh"
Sep 18 14:24:16.383978 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:16.383952    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c8c05d0-dc2f-494d-92a1-971574ddd421-istio-podinfo\") pod \"5c8c05d0-dc2f-494d-92a1-971574ddd421\" (UID: \"5c8c05d0-dc2f-494d-92a1-971574ddd421\") "
Sep 18 14:24:16.384159 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:16.383987    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c8c05d0-dc2f-494d-92a1-971574ddd421-istio-data\") pod \"5c8c05d0-dc2f-494d-92a1-971574ddd421\" (UID: \"5c8c05d0-dc2f-494d-92a1-971574ddd421\") "
Sep 18 14:24:16.384159 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:16.384002    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c8c05d0-dc2f-494d-92a1-971574ddd421-istio-envoy\") pod \"5c8c05d0-dc2f-494d-92a1-971574ddd421\" (UID: \"5c8c05d0-dc2f-494d-92a1-971574ddd421\") "
Sep 18 14:24:16.384159 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:16.384033    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fckr\" (UniqueName: \"kubernetes.io/projected/5c8c05d0-dc2f-494d-92a1-971574ddd421-kube-api-access-8fckr\") pod \"5c8c05d0-dc2f-494d-92a1-971574ddd421\" (UID: \"5c8c05d0-dc2f-494d-92a1-971574ddd421\") "
Sep 18 14:24:16.384159 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:16.384076    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c8c05d0-dc2f-494d-92a1-971574ddd421-workload-certs\") pod \"5c8c05d0-dc2f-494d-92a1-971574ddd421\" (UID: \"5c8c05d0-dc2f-494d-92a1-971574ddd421\") "
Sep 18 14:24:16.384159 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:16.384099    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c8c05d0-dc2f-494d-92a1-971574ddd421-credential-socket\") pod \"5c8c05d0-dc2f-494d-92a1-971574ddd421\" (UID: \"5c8c05d0-dc2f-494d-92a1-971574ddd421\") "
Sep 18 14:24:16.384487 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:16.384191    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c8c05d0-dc2f-494d-92a1-971574ddd421-istiod-ca-cert\") pod \"5c8c05d0-dc2f-494d-92a1-971574ddd421\" (UID: \"5c8c05d0-dc2f-494d-92a1-971574ddd421\") "
Sep 18 14:24:16.384487 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:16.384222    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c8c05d0-dc2f-494d-92a1-971574ddd421-workload-socket\") pod \"5c8c05d0-dc2f-494d-92a1-971574ddd421\" (UID: \"5c8c05d0-dc2f-494d-92a1-971574ddd421\") "
Sep 18 14:24:16.384487 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:16.384264    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c8c05d0-dc2f-494d-92a1-971574ddd421-istio-token\") pod \"5c8c05d0-dc2f-494d-92a1-971574ddd421\" (UID: \"5c8c05d0-dc2f-494d-92a1-971574ddd421\") "
Sep 18 14:24:16.384636 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:16.384260    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c8c05d0-dc2f-494d-92a1-971574ddd421-istio-data" (OuterVolumeSpecName: "istio-data") pod "5c8c05d0-dc2f-494d-92a1-971574ddd421" (UID: "5c8c05d0-dc2f-494d-92a1-971574ddd421"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:24:16.384636 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:16.384396    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c8c05d0-dc2f-494d-92a1-971574ddd421-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5c8c05d0-dc2f-494d-92a1-971574ddd421" (UID: "5c8c05d0-dc2f-494d-92a1-971574ddd421"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:24:16.384636 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:16.384444    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c8c05d0-dc2f-494d-92a1-971574ddd421-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5c8c05d0-dc2f-494d-92a1-971574ddd421" (UID: "5c8c05d0-dc2f-494d-92a1-971574ddd421"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:24:16.384799 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:16.384614    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c8c05d0-dc2f-494d-92a1-971574ddd421-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:24:16.384799 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:16.384668    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c8c05d0-dc2f-494d-92a1-971574ddd421-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:24:16.384799 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:16.384683    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c8c05d0-dc2f-494d-92a1-971574ddd421-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:24:16.384799 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:16.384701    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c8c05d0-dc2f-494d-92a1-971574ddd421-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5c8c05d0-dc2f-494d-92a1-971574ddd421" (UID: "5c8c05d0-dc2f-494d-92a1-971574ddd421"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:24:16.384799 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:16.384774    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c8c05d0-dc2f-494d-92a1-971574ddd421-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5c8c05d0-dc2f-494d-92a1-971574ddd421" (UID: "5c8c05d0-dc2f-494d-92a1-971574ddd421"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:24:16.386342 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:16.386317    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c8c05d0-dc2f-494d-92a1-971574ddd421-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5c8c05d0-dc2f-494d-92a1-971574ddd421" (UID: "5c8c05d0-dc2f-494d-92a1-971574ddd421"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 14:24:16.386456 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:16.386416    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c8c05d0-dc2f-494d-92a1-971574ddd421-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5c8c05d0-dc2f-494d-92a1-971574ddd421" (UID: "5c8c05d0-dc2f-494d-92a1-971574ddd421"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:24:16.386456 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:16.386424    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c8c05d0-dc2f-494d-92a1-971574ddd421-istio-token" (OuterVolumeSpecName: "istio-token") pod "5c8c05d0-dc2f-494d-92a1-971574ddd421" (UID: "5c8c05d0-dc2f-494d-92a1-971574ddd421"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:24:16.386545 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:16.386477    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c8c05d0-dc2f-494d-92a1-971574ddd421-kube-api-access-8fckr" (OuterVolumeSpecName: "kube-api-access-8fckr") pod "5c8c05d0-dc2f-494d-92a1-971574ddd421" (UID: "5c8c05d0-dc2f-494d-92a1-971574ddd421"). InnerVolumeSpecName "kube-api-access-8fckr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:24:16.485015 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:16.484990    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c8c05d0-dc2f-494d-92a1-971574ddd421-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:24:16.485015 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:16.485012    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c8c05d0-dc2f-494d-92a1-971574ddd421-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:24:16.485156 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:16.485021    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fckr\" (UniqueName: \"kubernetes.io/projected/5c8c05d0-dc2f-494d-92a1-971574ddd421-kube-api-access-8fckr\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:24:16.485156 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:16.485031    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c8c05d0-dc2f-494d-92a1-971574ddd421-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:24:16.485156 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:16.485038    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c8c05d0-dc2f-494d-92a1-971574ddd421-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:24:16.485156 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:16.485046    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c8c05d0-dc2f-494d-92a1-971574ddd421-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:24:16.562465 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:16.562438    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:17.060946 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:17.060917    2579 generic.go:356] "Generic (PLEG): container finished" podID="5c8c05d0-dc2f-494d-92a1-971574ddd421" containerID="d0da1fd7051779ba08032032d6889b0ce3fdbb1a54e240dc84166af2759ec276" exitCode=0
Sep 18 14:24:17.061108 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:17.060968    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh"
Sep 18 14:24:17.061108 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:17.061009    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh" event={"ID":"5c8c05d0-dc2f-494d-92a1-971574ddd421","Type":"ContainerDied","Data":"d0da1fd7051779ba08032032d6889b0ce3fdbb1a54e240dc84166af2759ec276"}
Sep 18 14:24:17.061108 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:17.061065    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh" event={"ID":"5c8c05d0-dc2f-494d-92a1-971574ddd421","Type":"ContainerDied","Data":"906803a771b8aee323d7cebdd1ea584b5690734ddd227071131f2a4aea3f18d6"}
Sep 18 14:24:17.061108 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:17.061095    2579 scope.go:117] "RemoveContainer" containerID="d0da1fd7051779ba08032032d6889b0ce3fdbb1a54e240dc84166af2759ec276"
Sep 18 14:24:17.070880 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:17.070863    2579 scope.go:117] "RemoveContainer" containerID="4228653cddd5a7a7ef3c4a60c53cd22a331c4d55bf6aaf1fbe72183aa77208a8"
Sep 18 14:24:17.078573 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:17.078556    2579 scope.go:117] "RemoveContainer" containerID="d0da1fd7051779ba08032032d6889b0ce3fdbb1a54e240dc84166af2759ec276"
Sep 18 14:24:17.078807 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:24:17.078788    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0da1fd7051779ba08032032d6889b0ce3fdbb1a54e240dc84166af2759ec276\": container with ID starting with d0da1fd7051779ba08032032d6889b0ce3fdbb1a54e240dc84166af2759ec276 not found: ID does not exist" containerID="d0da1fd7051779ba08032032d6889b0ce3fdbb1a54e240dc84166af2759ec276"
Sep 18 14:24:17.078860 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:17.078815    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0da1fd7051779ba08032032d6889b0ce3fdbb1a54e240dc84166af2759ec276"} err="failed to get container status \"d0da1fd7051779ba08032032d6889b0ce3fdbb1a54e240dc84166af2759ec276\": rpc error: code = NotFound desc = could not find container \"d0da1fd7051779ba08032032d6889b0ce3fdbb1a54e240dc84166af2759ec276\": container with ID starting with d0da1fd7051779ba08032032d6889b0ce3fdbb1a54e240dc84166af2759ec276 not found: ID does not exist"
Sep 18 14:24:17.078860 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:17.078832    2579 scope.go:117] "RemoveContainer" containerID="4228653cddd5a7a7ef3c4a60c53cd22a331c4d55bf6aaf1fbe72183aa77208a8"
Sep 18 14:24:17.079063 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:24:17.079046    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4228653cddd5a7a7ef3c4a60c53cd22a331c4d55bf6aaf1fbe72183aa77208a8\": container with ID starting with 4228653cddd5a7a7ef3c4a60c53cd22a331c4d55bf6aaf1fbe72183aa77208a8 not found: ID does not exist" containerID="4228653cddd5a7a7ef3c4a60c53cd22a331c4d55bf6aaf1fbe72183aa77208a8"
Sep 18 14:24:17.079102 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:17.079070    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4228653cddd5a7a7ef3c4a60c53cd22a331c4d55bf6aaf1fbe72183aa77208a8"} err="failed to get container status \"4228653cddd5a7a7ef3c4a60c53cd22a331c4d55bf6aaf1fbe72183aa77208a8\": rpc error: code = NotFound desc = could not find container \"4228653cddd5a7a7ef3c4a60c53cd22a331c4d55bf6aaf1fbe72183aa77208a8\": container with ID starting with 4228653cddd5a7a7ef3c4a60c53cd22a331c4d55bf6aaf1fbe72183aa77208a8 not found: ID does not exist"
Sep 18 14:24:17.084319 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:17.084294    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh"]
Sep 18 14:24:17.092264 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:17.092243    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-e01050-openshift-default-74cdc89bdb-hp8nh"]
Sep 18 14:24:17.976852 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:17.976817    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:18.035779 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:18.035750    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c8c05d0-dc2f-494d-92a1-971574ddd421" path="/var/lib/kubelet/pods/5c8c05d0-dc2f-494d-92a1-971574ddd421/volumes"
Sep 18 14:24:27.000773 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:27.000737    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:27.102954 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:27.102922    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:27.206161 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:27.206119    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:27.285343 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:27.285260    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:27.620473 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:27.620434    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:27.749541 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:27.749505    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:28.434328 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:28.434285    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:29.126373 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:29.126326    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:29.975369 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:29.975333    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:30.717197 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:30.717143    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:34.337260 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:34.337217    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:34.406885 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:34.406844    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:34.583944 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:34.583910    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:35.161408 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:35.161375    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:35.980312 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:35.980284    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:36.800146 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:36.800115    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:37.878490 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:37.878449    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:38.817535 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:38.817503    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:39.644521 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:39.644478    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:45.411310 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:45.411278    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:45.543529 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:45.543494    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:45.640728 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:45.640690    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:45.724683 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:45.724596    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:46.157632 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:46.157592    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:46.252301 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:46.252273    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:47.156673 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:47.156646    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:48.242905 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:48.242872    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:49.303616 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:49.303587    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:52.096329 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:52.096298    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:52.221105 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:52.221076    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:52.342541 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:52.342511    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm"]
Sep 18 14:24:52.342754 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:52.342730    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm" podUID="58292587-7ef6-4a85-a3b2-fede8ba8ca4c" containerName="istio-proxy" containerID="cri-o://7e79e803447af61f33c8f2cffe7fe92c8cbd2f548066e149ee02abca503fa346" gracePeriod=30
Sep 18 14:24:52.369453 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:52.369373    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:52.987571 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:52.987538    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:53.971841 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:53.971810    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:54.951315 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:54.951287    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:55.050946 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:55.050917    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:55.818402 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:55.818370    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:55.954226 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:55.954137    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:56.294326 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:56.294236    2579 patch_prober.go:28] interesting pod/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 14:24:56.294326 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:56.294304    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm" podUID="58292587-7ef6-4a85-a3b2-fede8ba8ca4c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 14:24:56.481520 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:56.481475    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:24:57.622211 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:57.622163    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm"
Sep 18 14:24:57.753819 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:57.753784    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-istiod-ca-cert\") pod \"58292587-7ef6-4a85-a3b2-fede8ba8ca4c\" (UID: \"58292587-7ef6-4a85-a3b2-fede8ba8ca4c\") "
Sep 18 14:24:57.754006 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:57.753837    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6wl5s\" (UniqueName: \"kubernetes.io/projected/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-kube-api-access-6wl5s\") pod \"58292587-7ef6-4a85-a3b2-fede8ba8ca4c\" (UID: \"58292587-7ef6-4a85-a3b2-fede8ba8ca4c\") "
Sep 18 14:24:57.754006 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:57.753867    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-istio-envoy\") pod \"58292587-7ef6-4a85-a3b2-fede8ba8ca4c\" (UID: \"58292587-7ef6-4a85-a3b2-fede8ba8ca4c\") "
Sep 18 14:24:57.754006 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:57.753972    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-istio-podinfo\") pod \"58292587-7ef6-4a85-a3b2-fede8ba8ca4c\" (UID: \"58292587-7ef6-4a85-a3b2-fede8ba8ca4c\") "
Sep 18 14:24:57.754213 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:57.754028    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-workload-certs\") pod \"58292587-7ef6-4a85-a3b2-fede8ba8ca4c\" (UID: \"58292587-7ef6-4a85-a3b2-fede8ba8ca4c\") "
Sep 18 14:24:57.754213 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:57.754072    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-credential-socket\") pod \"58292587-7ef6-4a85-a3b2-fede8ba8ca4c\" (UID: \"58292587-7ef6-4a85-a3b2-fede8ba8ca4c\") "
Sep 18 14:24:57.754213 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:57.754102    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-workload-socket\") pod \"58292587-7ef6-4a85-a3b2-fede8ba8ca4c\" (UID: \"58292587-7ef6-4a85-a3b2-fede8ba8ca4c\") "
Sep 18 14:24:57.754213 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:57.754167    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-istio-token\") pod \"58292587-7ef6-4a85-a3b2-fede8ba8ca4c\" (UID: \"58292587-7ef6-4a85-a3b2-fede8ba8ca4c\") "
Sep 18 14:24:57.754425 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:57.754210    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "58292587-7ef6-4a85-a3b2-fede8ba8ca4c" (UID: "58292587-7ef6-4a85-a3b2-fede8ba8ca4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:24:57.754425 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:57.754261    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-istio-data\") pod \"58292587-7ef6-4a85-a3b2-fede8ba8ca4c\" (UID: \"58292587-7ef6-4a85-a3b2-fede8ba8ca4c\") "
Sep 18 14:24:57.754612 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:57.754594    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:24:57.754703 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:57.754630    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "58292587-7ef6-4a85-a3b2-fede8ba8ca4c" (UID: "58292587-7ef6-4a85-a3b2-fede8ba8ca4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:24:57.754703 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:57.754689    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "58292587-7ef6-4a85-a3b2-fede8ba8ca4c" (UID: "58292587-7ef6-4a85-a3b2-fede8ba8ca4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:24:57.754818 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:57.754794    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "58292587-7ef6-4a85-a3b2-fede8ba8ca4c" (UID: "58292587-7ef6-4a85-a3b2-fede8ba8ca4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:24:57.754874 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:57.754814    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-istio-data" (OuterVolumeSpecName: "istio-data") pod "58292587-7ef6-4a85-a3b2-fede8ba8ca4c" (UID: "58292587-7ef6-4a85-a3b2-fede8ba8ca4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:24:57.756477 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:57.756448    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "58292587-7ef6-4a85-a3b2-fede8ba8ca4c" (UID: "58292587-7ef6-4a85-a3b2-fede8ba8ca4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:24:57.756605 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:57.756545    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-kube-api-access-6wl5s" (OuterVolumeSpecName: "kube-api-access-6wl5s") pod "58292587-7ef6-4a85-a3b2-fede8ba8ca4c" (UID: "58292587-7ef6-4a85-a3b2-fede8ba8ca4c"). InnerVolumeSpecName "kube-api-access-6wl5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:24:57.756938 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:57.756914    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-istio-token" (OuterVolumeSpecName: "istio-token") pod "58292587-7ef6-4a85-a3b2-fede8ba8ca4c" (UID: "58292587-7ef6-4a85-a3b2-fede8ba8ca4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:24:57.756990 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:57.756915    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "58292587-7ef6-4a85-a3b2-fede8ba8ca4c" (UID: "58292587-7ef6-4a85-a3b2-fede8ba8ca4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 14:24:57.855195 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:57.855121    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:24:57.855195 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:57.855145    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wl5s\" (UniqueName: \"kubernetes.io/projected/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-kube-api-access-6wl5s\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:24:57.855195 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:57.855157    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:24:57.855195 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:57.855181    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:24:57.855195 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:57.855197    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:24:57.855415 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:57.855209    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:24:57.855415 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:57.855220    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:24:57.855415 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:57.855231    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58292587-7ef6-4a85-a3b2-fede8ba8ca4c-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:24:58.271784 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:58.271716    2579 generic.go:356] "Generic (PLEG): container finished" podID="58292587-7ef6-4a85-a3b2-fede8ba8ca4c" containerID="7e79e803447af61f33c8f2cffe7fe92c8cbd2f548066e149ee02abca503fa346" exitCode=0
Sep 18 14:24:58.271784 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:58.271769    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm"
Sep 18 14:24:58.271928 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:58.271788    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm" event={"ID":"58292587-7ef6-4a85-a3b2-fede8ba8ca4c","Type":"ContainerDied","Data":"7e79e803447af61f33c8f2cffe7fe92c8cbd2f548066e149ee02abca503fa346"}
Sep 18 14:24:58.271928 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:58.271832    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm" event={"ID":"58292587-7ef6-4a85-a3b2-fede8ba8ca4c","Type":"ContainerDied","Data":"eda36fa87d6e3d1827a1e0f25da10625dd58777511e10aed864b87e20aa7138d"}
Sep 18 14:24:58.271928 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:58.271854    2579 scope.go:117] "RemoveContainer" containerID="7e79e803447af61f33c8f2cffe7fe92c8cbd2f548066e149ee02abca503fa346"
Sep 18 14:24:58.280948 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:58.280916    2579 scope.go:117] "RemoveContainer" containerID="fffe684e9482ececbe3ba677486bf3e32c2ff62ffad6d57cbac25c9cd7ddf1ad"
Sep 18 14:24:58.288399 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:58.288381    2579 scope.go:117] "RemoveContainer" containerID="7e79e803447af61f33c8f2cffe7fe92c8cbd2f548066e149ee02abca503fa346"
Sep 18 14:24:58.288660 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:24:58.288641    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e79e803447af61f33c8f2cffe7fe92c8cbd2f548066e149ee02abca503fa346\": container with ID starting with 7e79e803447af61f33c8f2cffe7fe92c8cbd2f548066e149ee02abca503fa346 not found: ID does not exist" containerID="7e79e803447af61f33c8f2cffe7fe92c8cbd2f548066e149ee02abca503fa346"
Sep 18 14:24:58.288727 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:58.288667    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e79e803447af61f33c8f2cffe7fe92c8cbd2f548066e149ee02abca503fa346"} err="failed to get container status \"7e79e803447af61f33c8f2cffe7fe92c8cbd2f548066e149ee02abca503fa346\": rpc error: code = NotFound desc = could not find container \"7e79e803447af61f33c8f2cffe7fe92c8cbd2f548066e149ee02abca503fa346\": container with ID starting with 7e79e803447af61f33c8f2cffe7fe92c8cbd2f548066e149ee02abca503fa346 not found: ID does not exist"
Sep 18 14:24:58.288727 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:58.288683    2579 scope.go:117] "RemoveContainer" containerID="fffe684e9482ececbe3ba677486bf3e32c2ff62ffad6d57cbac25c9cd7ddf1ad"
Sep 18 14:24:58.288955 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:24:58.288936    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fffe684e9482ececbe3ba677486bf3e32c2ff62ffad6d57cbac25c9cd7ddf1ad\": container with ID starting with fffe684e9482ececbe3ba677486bf3e32c2ff62ffad6d57cbac25c9cd7ddf1ad not found: ID does not exist" containerID="fffe684e9482ececbe3ba677486bf3e32c2ff62ffad6d57cbac25c9cd7ddf1ad"
Sep 18 14:24:58.289019 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:58.288961    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fffe684e9482ececbe3ba677486bf3e32c2ff62ffad6d57cbac25c9cd7ddf1ad"} err="failed to get container status \"fffe684e9482ececbe3ba677486bf3e32c2ff62ffad6d57cbac25c9cd7ddf1ad\": rpc error: code = NotFound desc = could not find container \"fffe684e9482ececbe3ba677486bf3e32c2ff62ffad6d57cbac25c9cd7ddf1ad\": container with ID starting with fffe684e9482ececbe3ba677486bf3e32c2ff62ffad6d57cbac25c9cd7ddf1ad not found: ID does not exist"
Sep 18 14:24:58.289862 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:58.289840    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm"]
Sep 18 14:24:58.293286 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:58.293265    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-526b97-openshift-default-7dfbd688d-mvfjm"]
Sep 18 14:24:59.172532 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:24:59.172499    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:25:00.035444 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:00.035414    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58292587-7ef6-4a85-a3b2-fede8ba8ca4c" path="/var/lib/kubelet/pods/58292587-7ef6-4a85-a3b2-fede8ba8ca4c/volumes"
Sep 18 14:25:03.242110 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:03.242075    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-216d39-6449979c-28lww"]
Sep 18 14:25:03.242576 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:03.242308    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-216d39-6449979c-28lww" podUID="e987aa89-8e53-41ef-9ab3-48bb6d51ca52" containerName="maas-api" containerID="cri-o://ace33f630457bacd6bd4ce248340d6ffb17d7fb81ed1357e217314f8e7aa0203" gracePeriod=30
Sep 18 14:25:03.277180 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:03.277139    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:25:03.346514 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:03.346484    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:25:03.415649 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:03.415417    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:25:03.492466 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:03.492409    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-216d39-6449979c-28lww"
Sep 18 14:25:03.493597 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:03.493575    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:25:03.602125 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:03.602100    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4n88w\" (UniqueName: \"kubernetes.io/projected/e987aa89-8e53-41ef-9ab3-48bb6d51ca52-kube-api-access-4n88w\") pod \"e987aa89-8e53-41ef-9ab3-48bb6d51ca52\" (UID: \"e987aa89-8e53-41ef-9ab3-48bb6d51ca52\") "
Sep 18 14:25:03.602293 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:03.602142    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e987aa89-8e53-41ef-9ab3-48bb6d51ca52-maas-api-tls\") pod \"e987aa89-8e53-41ef-9ab3-48bb6d51ca52\" (UID: \"e987aa89-8e53-41ef-9ab3-48bb6d51ca52\") "
Sep 18 14:25:03.602293 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:03.602283    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e987aa89-8e53-41ef-9ab3-48bb6d51ca52-openshift-service-ca\") pod \"e987aa89-8e53-41ef-9ab3-48bb6d51ca52\" (UID: \"e987aa89-8e53-41ef-9ab3-48bb6d51ca52\") "
Sep 18 14:25:03.602426 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:03.602347    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e987aa89-8e53-41ef-9ab3-48bb6d51ca52-metrics-certs\") pod \"e987aa89-8e53-41ef-9ab3-48bb6d51ca52\" (UID: \"e987aa89-8e53-41ef-9ab3-48bb6d51ca52\") "
Sep 18 14:25:03.602636 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:03.602614    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e987aa89-8e53-41ef-9ab3-48bb6d51ca52-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e987aa89-8e53-41ef-9ab3-48bb6d51ca52" (UID: "e987aa89-8e53-41ef-9ab3-48bb6d51ca52"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:25:03.604391 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:03.604370    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e987aa89-8e53-41ef-9ab3-48bb6d51ca52-kube-api-access-4n88w" (OuterVolumeSpecName: "kube-api-access-4n88w") pod "e987aa89-8e53-41ef-9ab3-48bb6d51ca52" (UID: "e987aa89-8e53-41ef-9ab3-48bb6d51ca52"). InnerVolumeSpecName "kube-api-access-4n88w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:25:03.604667 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:03.604631    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e987aa89-8e53-41ef-9ab3-48bb6d51ca52-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e987aa89-8e53-41ef-9ab3-48bb6d51ca52" (UID: "e987aa89-8e53-41ef-9ab3-48bb6d51ca52"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:25:03.604667 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:03.604651    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e987aa89-8e53-41ef-9ab3-48bb6d51ca52-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e987aa89-8e53-41ef-9ab3-48bb6d51ca52" (UID: "e987aa89-8e53-41ef-9ab3-48bb6d51ca52"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:25:03.704000 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:03.703958    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n88w\" (UniqueName: \"kubernetes.io/projected/e987aa89-8e53-41ef-9ab3-48bb6d51ca52-kube-api-access-4n88w\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:25:03.704000 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:03.703990    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e987aa89-8e53-41ef-9ab3-48bb6d51ca52-maas-api-tls\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:25:03.704000 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:03.704000    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e987aa89-8e53-41ef-9ab3-48bb6d51ca52-openshift-service-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:25:03.704000 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:03.704010    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e987aa89-8e53-41ef-9ab3-48bb6d51ca52-metrics-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:25:03.844380 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:03.844346    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:25:03.948466 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:03.948442    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:25:04.302448 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:04.302419    2579 generic.go:356] "Generic (PLEG): container finished" podID="e987aa89-8e53-41ef-9ab3-48bb6d51ca52" containerID="ace33f630457bacd6bd4ce248340d6ffb17d7fb81ed1357e217314f8e7aa0203" exitCode=0
Sep 18 14:25:04.302822 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:04.302456    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-216d39-6449979c-28lww"
Sep 18 14:25:04.302822 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:04.302501    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-216d39-6449979c-28lww" event={"ID":"e987aa89-8e53-41ef-9ab3-48bb6d51ca52","Type":"ContainerDied","Data":"ace33f630457bacd6bd4ce248340d6ffb17d7fb81ed1357e217314f8e7aa0203"}
Sep 18 14:25:04.302822 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:04.302553    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-216d39-6449979c-28lww" event={"ID":"e987aa89-8e53-41ef-9ab3-48bb6d51ca52","Type":"ContainerDied","Data":"edd5660f17487cd8c03b2af496f3d72d3eaee5a297ba9bc85ebdfe9753f3fdd8"}
Sep 18 14:25:04.302822 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:04.302576    2579 scope.go:117] "RemoveContainer" containerID="ace33f630457bacd6bd4ce248340d6ffb17d7fb81ed1357e217314f8e7aa0203"
Sep 18 14:25:04.310876 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:04.310860    2579 scope.go:117] "RemoveContainer" containerID="ace33f630457bacd6bd4ce248340d6ffb17d7fb81ed1357e217314f8e7aa0203"
Sep 18 14:25:04.311146 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:25:04.311128    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ace33f630457bacd6bd4ce248340d6ffb17d7fb81ed1357e217314f8e7aa0203\": container with ID starting with ace33f630457bacd6bd4ce248340d6ffb17d7fb81ed1357e217314f8e7aa0203 not found: ID does not exist" containerID="ace33f630457bacd6bd4ce248340d6ffb17d7fb81ed1357e217314f8e7aa0203"
Sep 18 14:25:04.311226 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:04.311158    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ace33f630457bacd6bd4ce248340d6ffb17d7fb81ed1357e217314f8e7aa0203"} err="failed to get container status \"ace33f630457bacd6bd4ce248340d6ffb17d7fb81ed1357e217314f8e7aa0203\": rpc error: code = NotFound desc = could not find container \"ace33f630457bacd6bd4ce248340d6ffb17d7fb81ed1357e217314f8e7aa0203\": container with ID starting with ace33f630457bacd6bd4ce248340d6ffb17d7fb81ed1357e217314f8e7aa0203 not found: ID does not exist"
Sep 18 14:25:04.323701 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:04.323676    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-216d39-6449979c-28lww"]
Sep 18 14:25:04.328029 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:04.328008    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-216d39-6449979c-28lww"]
Sep 18 14:25:04.814749 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:04.814716    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:25:06.035297 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:06.035265    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e987aa89-8e53-41ef-9ab3-48bb6d51ca52" path="/var/lib/kubelet/pods/e987aa89-8e53-41ef-9ab3-48bb6d51ca52/volumes"
Sep 18 14:25:06.692062 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:06.692033    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:25:09.810289 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:09.810252    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:25:09.881793 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:09.881765    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:25:10.025327 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:10.025297    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst"]
Sep 18 14:25:10.025564 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:10.025536    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst" podUID="4978be46-34db-43ce-87e2-a9c3bd6df751" containerName="istio-proxy" containerID="cri-o://897d743f2e731560531a67d83ffcfb8f80f8b1e859a332c3fce92116987b1938" gracePeriod=30
Sep 18 14:25:10.049732 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:10.049699    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:25:10.764797 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:10.764765    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 14:25:10.764993 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:10.764836    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst" podUID="4978be46-34db-43ce-87e2-a9c3bd6df751" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 14:25:15.284821 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:15.284794    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst"
Sep 18 14:25:15.360889 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:15.360856    2579 generic.go:356] "Generic (PLEG): container finished" podID="4978be46-34db-43ce-87e2-a9c3bd6df751" containerID="897d743f2e731560531a67d83ffcfb8f80f8b1e859a332c3fce92116987b1938" exitCode=0
Sep 18 14:25:15.361048 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:15.360905    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst"
Sep 18 14:25:15.361048 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:15.360903    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst" event={"ID":"4978be46-34db-43ce-87e2-a9c3bd6df751","Type":"ContainerDied","Data":"897d743f2e731560531a67d83ffcfb8f80f8b1e859a332c3fce92116987b1938"}
Sep 18 14:25:15.361048 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:15.360960    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst" event={"ID":"4978be46-34db-43ce-87e2-a9c3bd6df751","Type":"ContainerDied","Data":"e7f09316976e6cb39771f23c7e19a0bb31a33a418ddcc5c3bc57f57cc45f6e8d"}
Sep 18 14:25:15.361048 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:15.360981    2579 scope.go:117] "RemoveContainer" containerID="897d743f2e731560531a67d83ffcfb8f80f8b1e859a332c3fce92116987b1938"
Sep 18 14:25:15.369184 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:15.369158    2579 scope.go:117] "RemoveContainer" containerID="ef3fce4f25faafd3318637d228f2b2503f313c67e091d4af2e3641d381a8cafa"
Sep 18 14:25:15.376328 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:15.376312    2579 scope.go:117] "RemoveContainer" containerID="897d743f2e731560531a67d83ffcfb8f80f8b1e859a332c3fce92116987b1938"
Sep 18 14:25:15.376595 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:25:15.376576    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"897d743f2e731560531a67d83ffcfb8f80f8b1e859a332c3fce92116987b1938\": container with ID starting with 897d743f2e731560531a67d83ffcfb8f80f8b1e859a332c3fce92116987b1938 not found: ID does not exist" containerID="897d743f2e731560531a67d83ffcfb8f80f8b1e859a332c3fce92116987b1938"
Sep 18 14:25:15.376653 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:15.376603    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"897d743f2e731560531a67d83ffcfb8f80f8b1e859a332c3fce92116987b1938"} err="failed to get container status \"897d743f2e731560531a67d83ffcfb8f80f8b1e859a332c3fce92116987b1938\": rpc error: code = NotFound desc = could not find container \"897d743f2e731560531a67d83ffcfb8f80f8b1e859a332c3fce92116987b1938\": container with ID starting with 897d743f2e731560531a67d83ffcfb8f80f8b1e859a332c3fce92116987b1938 not found: ID does not exist"
Sep 18 14:25:15.376653 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:15.376619    2579 scope.go:117] "RemoveContainer" containerID="ef3fce4f25faafd3318637d228f2b2503f313c67e091d4af2e3641d381a8cafa"
Sep 18 14:25:15.376855 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:25:15.376841    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef3fce4f25faafd3318637d228f2b2503f313c67e091d4af2e3641d381a8cafa\": container with ID starting with ef3fce4f25faafd3318637d228f2b2503f313c67e091d4af2e3641d381a8cafa not found: ID does not exist" containerID="ef3fce4f25faafd3318637d228f2b2503f313c67e091d4af2e3641d381a8cafa"
Sep 18 14:25:15.376903 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:15.376858    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef3fce4f25faafd3318637d228f2b2503f313c67e091d4af2e3641d381a8cafa"} err="failed to get container status \"ef3fce4f25faafd3318637d228f2b2503f313c67e091d4af2e3641d381a8cafa\": rpc error: code = NotFound desc = could not find container \"ef3fce4f25faafd3318637d228f2b2503f313c67e091d4af2e3641d381a8cafa\": container with ID starting with ef3fce4f25faafd3318637d228f2b2503f313c67e091d4af2e3641d381a8cafa not found: ID does not exist"
Sep 18 14:25:15.397294 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:15.397264    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4978be46-34db-43ce-87e2-a9c3bd6df751-istiod-ca-cert\") pod \"4978be46-34db-43ce-87e2-a9c3bd6df751\" (UID: \"4978be46-34db-43ce-87e2-a9c3bd6df751\") "
Sep 18 14:25:15.397386 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:15.397310    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4978be46-34db-43ce-87e2-a9c3bd6df751-istio-podinfo\") pod \"4978be46-34db-43ce-87e2-a9c3bd6df751\" (UID: \"4978be46-34db-43ce-87e2-a9c3bd6df751\") "
Sep 18 14:25:15.397386 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:15.397332    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4978be46-34db-43ce-87e2-a9c3bd6df751-workload-certs\") pod \"4978be46-34db-43ce-87e2-a9c3bd6df751\" (UID: \"4978be46-34db-43ce-87e2-a9c3bd6df751\") "
Sep 18 14:25:15.397386 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:15.397374    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zdqqp\" (UniqueName: \"kubernetes.io/projected/4978be46-34db-43ce-87e2-a9c3bd6df751-kube-api-access-zdqqp\") pod \"4978be46-34db-43ce-87e2-a9c3bd6df751\" (UID: \"4978be46-34db-43ce-87e2-a9c3bd6df751\") "
Sep 18 14:25:15.397544 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:15.397402    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4978be46-34db-43ce-87e2-a9c3bd6df751-istio-envoy\") pod \"4978be46-34db-43ce-87e2-a9c3bd6df751\" (UID: \"4978be46-34db-43ce-87e2-a9c3bd6df751\") "
Sep 18 14:25:15.397544 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:15.397478    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4978be46-34db-43ce-87e2-a9c3bd6df751-credential-socket\") pod \"4978be46-34db-43ce-87e2-a9c3bd6df751\" (UID: \"4978be46-34db-43ce-87e2-a9c3bd6df751\") "
Sep 18 14:25:15.397544 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:15.397533    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4978be46-34db-43ce-87e2-a9c3bd6df751-istio-token\") pod \"4978be46-34db-43ce-87e2-a9c3bd6df751\" (UID: \"4978be46-34db-43ce-87e2-a9c3bd6df751\") "
Sep 18 14:25:15.397697 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:15.397574    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4978be46-34db-43ce-87e2-a9c3bd6df751-istio-data\") pod \"4978be46-34db-43ce-87e2-a9c3bd6df751\" (UID: \"4978be46-34db-43ce-87e2-a9c3bd6df751\") "
Sep 18 14:25:15.397697 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:15.397603    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4978be46-34db-43ce-87e2-a9c3bd6df751-workload-socket\") pod \"4978be46-34db-43ce-87e2-a9c3bd6df751\" (UID: \"4978be46-34db-43ce-87e2-a9c3bd6df751\") "
Sep 18 14:25:15.397697 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:15.397644    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4978be46-34db-43ce-87e2-a9c3bd6df751-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4978be46-34db-43ce-87e2-a9c3bd6df751" (UID: "4978be46-34db-43ce-87e2-a9c3bd6df751"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:25:15.397697 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:15.397654    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4978be46-34db-43ce-87e2-a9c3bd6df751-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4978be46-34db-43ce-87e2-a9c3bd6df751" (UID: "4978be46-34db-43ce-87e2-a9c3bd6df751"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:25:15.397917 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:15.397813    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4978be46-34db-43ce-87e2-a9c3bd6df751-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4978be46-34db-43ce-87e2-a9c3bd6df751" (UID: "4978be46-34db-43ce-87e2-a9c3bd6df751"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:25:15.397917 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:15.397879    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4978be46-34db-43ce-87e2-a9c3bd6df751-istio-data" (OuterVolumeSpecName: "istio-data") pod "4978be46-34db-43ce-87e2-a9c3bd6df751" (UID: "4978be46-34db-43ce-87e2-a9c3bd6df751"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:25:15.397917 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:15.397895    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4978be46-34db-43ce-87e2-a9c3bd6df751-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:25:15.397917 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:15.397914    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4978be46-34db-43ce-87e2-a9c3bd6df751-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:25:15.398122 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:15.398031    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4978be46-34db-43ce-87e2-a9c3bd6df751-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4978be46-34db-43ce-87e2-a9c3bd6df751" (UID: "4978be46-34db-43ce-87e2-a9c3bd6df751"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:25:15.399506 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:15.399483    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4978be46-34db-43ce-87e2-a9c3bd6df751-istio-token" (OuterVolumeSpecName: "istio-token") pod "4978be46-34db-43ce-87e2-a9c3bd6df751" (UID: "4978be46-34db-43ce-87e2-a9c3bd6df751"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:25:15.399621 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:15.399557    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4978be46-34db-43ce-87e2-a9c3bd6df751-kube-api-access-zdqqp" (OuterVolumeSpecName: "kube-api-access-zdqqp") pod "4978be46-34db-43ce-87e2-a9c3bd6df751" (UID: "4978be46-34db-43ce-87e2-a9c3bd6df751"). InnerVolumeSpecName "kube-api-access-zdqqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:25:15.399807 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:15.399789    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4978be46-34db-43ce-87e2-a9c3bd6df751-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4978be46-34db-43ce-87e2-a9c3bd6df751" (UID: "4978be46-34db-43ce-87e2-a9c3bd6df751"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 14:25:15.399807 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:15.399788    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4978be46-34db-43ce-87e2-a9c3bd6df751-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4978be46-34db-43ce-87e2-a9c3bd6df751" (UID: "4978be46-34db-43ce-87e2-a9c3bd6df751"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:25:15.498963 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:15.498940    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4978be46-34db-43ce-87e2-a9c3bd6df751-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:25:15.498963 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:15.498960    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdqqp\" (UniqueName: \"kubernetes.io/projected/4978be46-34db-43ce-87e2-a9c3bd6df751-kube-api-access-zdqqp\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:25:15.499099 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:15.498970    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4978be46-34db-43ce-87e2-a9c3bd6df751-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:25:15.499099 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:15.498979    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4978be46-34db-43ce-87e2-a9c3bd6df751-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:25:15.499099 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:15.498988    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4978be46-34db-43ce-87e2-a9c3bd6df751-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:25:15.499099 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:15.498996    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4978be46-34db-43ce-87e2-a9c3bd6df751-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:25:15.499099 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:15.499004    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4978be46-34db-43ce-87e2-a9c3bd6df751-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:25:15.686906 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:15.686883    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst"]
Sep 18 14:25:15.692735 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:15.692717    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-216d39-openshift-default-5897964574-n6bst"]
Sep 18 14:25:16.035054 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:16.035001    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4978be46-34db-43ce-87e2-a9c3bd6df751" path="/var/lib/kubelet/pods/4978be46-34db-43ce-87e2-a9c3bd6df751/volumes"
Sep 18 14:25:18.560381 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:18.560341    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:25:40.367034 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:40.366958    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:25:40.424032 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:40.423999    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:25:50.245555 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:25:50.245517    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:26:32.088963 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:26:32.088930    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:26:32.824480 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:26:32.824452    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:26:42.652438 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:26:42.652404    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:27:09.859900 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:27:09.859864    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:27:46.412738 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:27:46.412700    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:27:47.344915 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:27:47.344885    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:27:47.771909 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:27:47.771828    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:27:48.083528 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:27:48.083494    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:27:48.240734 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:27:48.240693    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:27:50.406595 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:27:50.406567    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:27:51.037662 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:27:51.037622    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:27:51.209895 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:27:51.209866    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:27:51.309483 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:27:51.309454    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:27:51.421481 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:27:51.421451    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:27:52.112976 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:27:52.112946    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:27:53.051117 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:27:53.051089    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:27:53.150590 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:27:53.150558    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:27:53.251089 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:27:53.251061    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:27:53.522202 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:27:53.522101    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:27:53.604457 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:27:53.604422    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:28:10.004080 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:10.004049    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:28:14.815762 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:14.815719    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:28:15.988525 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:15.988489    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:28:16.322400 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:16.322362    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:28:16.740145 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:16.740046    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:28:16.980853 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:16.980819    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:28:17.086755 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:17.086727    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:28:17.185617 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:17.185584    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:28:17.285178 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:17.285151    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:28:18.237628 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:18.237600    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:28:18.986555 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:18.986530    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:28:19.934210 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:19.934183    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:28:20.053037 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:20.053006    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:28:20.150957 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:20.150927    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:28:20.740771 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:20.740742    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:28:21.484317 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:21.484266    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:28:22.267142 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:22.267104    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:28:23.036617 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:23.036590    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:28:23.138003 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:23.137964    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:28:23.375757 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:23.375725    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:28:23.503357 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:23.503328    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:28:23.603244 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:23.603214    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:28:23.618564 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:23.618531    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7ed186-6596bd9ff8-jd7tp"]
Sep 18 14:28:23.618869 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:23.618855    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4978be46-34db-43ce-87e2-a9c3bd6df751" containerName="istio-proxy"
Sep 18 14:28:23.618869 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:23.618869    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4978be46-34db-43ce-87e2-a9c3bd6df751" containerName="istio-proxy"
Sep 18 14:28:23.618978 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:23.618887    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c8c05d0-dc2f-494d-92a1-971574ddd421" containerName="istio-proxy"
Sep 18 14:28:23.618978 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:23.618892    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c8c05d0-dc2f-494d-92a1-971574ddd421" containerName="istio-proxy"
Sep 18 14:28:23.618978 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:23.618901    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c8c05d0-dc2f-494d-92a1-971574ddd421" containerName="istio-proxy"
Sep 18 14:28:23.618978 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:23.618906    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c8c05d0-dc2f-494d-92a1-971574ddd421" containerName="istio-proxy"
Sep 18 14:28:23.618978 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:23.618913    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d516717-4809-4020-af5b-3cb8478f112d" containerName="maas-api"
Sep 18 14:28:23.618978 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:23.618918    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d516717-4809-4020-af5b-3cb8478f112d" containerName="maas-api"
Sep 18 14:28:23.618978 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:23.618925    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58292587-7ef6-4a85-a3b2-fede8ba8ca4c" containerName="istio-proxy"
Sep 18 14:28:23.618978 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:23.618930    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="58292587-7ef6-4a85-a3b2-fede8ba8ca4c" containerName="istio-proxy"
Sep 18 14:28:23.618978 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:23.618936    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e987aa89-8e53-41ef-9ab3-48bb6d51ca52" containerName="maas-api"
Sep 18 14:28:23.618978 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:23.618941    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e987aa89-8e53-41ef-9ab3-48bb6d51ca52" containerName="maas-api"
Sep 18 14:28:23.618978 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:23.618950    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26c2ff1c-e079-40da-933c-44a76190b5c1" containerName="payload-pre-processing"
Sep 18 14:28:23.618978 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:23.618955    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="26c2ff1c-e079-40da-933c-44a76190b5c1" containerName="payload-pre-processing"
Sep 18 14:28:23.619563 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:23.619005    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d516717-4809-4020-af5b-3cb8478f112d" containerName="maas-api"
Sep 18 14:28:23.619563 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:23.619013    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="58292587-7ef6-4a85-a3b2-fede8ba8ca4c" containerName="istio-proxy"
Sep 18 14:28:23.619563 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:23.619019    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4978be46-34db-43ce-87e2-a9c3bd6df751" containerName="istio-proxy"
Sep 18 14:28:23.619563 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:23.619023    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="26c2ff1c-e079-40da-933c-44a76190b5c1" containerName="payload-pre-processing"
Sep 18 14:28:23.619563 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:23.619029    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c8c05d0-dc2f-494d-92a1-971574ddd421" containerName="istio-proxy"
Sep 18 14:28:23.619563 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:23.619035    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c8c05d0-dc2f-494d-92a1-971574ddd421" containerName="istio-proxy"
Sep 18 14:28:23.619563 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:23.619042    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e987aa89-8e53-41ef-9ab3-48bb6d51ca52" containerName="maas-api"
Sep 18 14:28:23.619563 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:23.619047    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="58292587-7ef6-4a85-a3b2-fede8ba8ca4c" containerName="istio-proxy"
Sep 18 14:28:23.622143 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:23.622124    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7ed186-6596bd9ff8-jd7tp"
Sep 18 14:28:23.624723 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:23.624698    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-7ed186-dockercfg-gb4rv\""
Sep 18 14:28:23.624872 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:23.624702    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-7ed186\""
Sep 18 14:28:23.625574 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:23.625552    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7ed186-6596bd9ff8-jd7tp"]
Sep 18 14:28:23.776651 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:23.776619    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46f30e53-2dbf-41f9-abd4-82701d9594fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7ed186-6596bd9ff8-jd7tp\" (UID: \"46f30e53-2dbf-41f9-abd4-82701d9594fa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7ed186-6596bd9ff8-jd7tp"
Sep 18 14:28:23.776826 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:23.776682    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kqk9\" (UniqueName: \"kubernetes.io/projected/46f30e53-2dbf-41f9-abd4-82701d9594fa-kube-api-access-9kqk9\") pod \"payload-pre-processing-e2e-ipp-b-7ed186-6596bd9ff8-jd7tp\" (UID: \"46f30e53-2dbf-41f9-abd4-82701d9594fa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7ed186-6596bd9ff8-jd7tp"
Sep 18 14:28:23.877562 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:23.877531    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kqk9\" (UniqueName: \"kubernetes.io/projected/46f30e53-2dbf-41f9-abd4-82701d9594fa-kube-api-access-9kqk9\") pod \"payload-pre-processing-e2e-ipp-b-7ed186-6596bd9ff8-jd7tp\" (UID: \"46f30e53-2dbf-41f9-abd4-82701d9594fa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7ed186-6596bd9ff8-jd7tp"
Sep 18 14:28:23.877703 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:23.877595    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46f30e53-2dbf-41f9-abd4-82701d9594fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7ed186-6596bd9ff8-jd7tp\" (UID: \"46f30e53-2dbf-41f9-abd4-82701d9594fa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7ed186-6596bd9ff8-jd7tp"
Sep 18 14:28:23.878268 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:23.878251    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46f30e53-2dbf-41f9-abd4-82701d9594fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7ed186-6596bd9ff8-jd7tp\" (UID: \"46f30e53-2dbf-41f9-abd4-82701d9594fa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7ed186-6596bd9ff8-jd7tp"
Sep 18 14:28:23.885536 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:23.885485    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kqk9\" (UniqueName: \"kubernetes.io/projected/46f30e53-2dbf-41f9-abd4-82701d9594fa-kube-api-access-9kqk9\") pod \"payload-pre-processing-e2e-ipp-b-7ed186-6596bd9ff8-jd7tp\" (UID: \"46f30e53-2dbf-41f9-abd4-82701d9594fa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7ed186-6596bd9ff8-jd7tp"
Sep 18 14:28:23.933556 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:23.933532    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7ed186-6596bd9ff8-jd7tp"
Sep 18 14:28:24.206962 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:24.206939    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7ed186-6596bd9ff8-jd7tp"]
Sep 18 14:28:24.209412 ip-10-0-143-120 kubenswrapper[2579]: W0918 14:28:24.209387    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46f30e53_2dbf_41f9_abd4_82701d9594fa.slice/crio-98aa76b93f754ffcbaddbf050306d6126b751b368d08bfce6d19aec067b6fd59 WatchSource:0}: Error finding container 98aa76b93f754ffcbaddbf050306d6126b751b368d08bfce6d19aec067b6fd59: Status 404 returned error can't find the container with id 98aa76b93f754ffcbaddbf050306d6126b751b368d08bfce6d19aec067b6fd59
Sep 18 14:28:24.219389 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:24.219364    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7ed186-6596bd9ff8-jd7tp" event={"ID":"46f30e53-2dbf-41f9-abd4-82701d9594fa","Type":"ContainerStarted","Data":"98aa76b93f754ffcbaddbf050306d6126b751b368d08bfce6d19aec067b6fd59"}
Sep 18 14:28:25.225278 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:25.225243    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7ed186-6596bd9ff8-jd7tp" event={"ID":"46f30e53-2dbf-41f9-abd4-82701d9594fa","Type":"ContainerStarted","Data":"76a14ca066398422b508bcd74083982061bd6672fb1e863aff0fcd9d9e563942"}
Sep 18 14:28:25.225651 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:25.225356    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7ed186-6596bd9ff8-jd7tp"
Sep 18 14:28:25.243072 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:25.243026    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7ed186-6596bd9ff8-jd7tp" podStartSLOduration=2.243013964 podStartE2EDuration="2.243013964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:28:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:28:25.240584935 +0000 UTC m=+1777.129300865" watchObservedRunningTime="2026-09-18 14:28:25.243013964 +0000 UTC m=+1777.131729946"
Sep 18 14:28:36.231540 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:36.231506    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7ed186-6596bd9ff8-jd7tp"
Sep 18 14:28:39.256443 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:39.256398    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:28:44.006569 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:44.006528    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:28:45.122384 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:45.122347    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:28:45.418037 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:45.417952    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:28:46.224597 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:46.224565    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:28:46.985348 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:46.985313    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:28:47.722361 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:47.722300    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:28:48.451701 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:48.451675    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:28:50.086225 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:50.086201    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sg4tv_4626a667-4cd6-468c-8ef3-6e3a925c2608/ovn-acl-logging/0.log"
Sep 18 14:28:50.090099 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:50.090077    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sg4tv_4626a667-4cd6-468c-8ef3-6e3a925c2608/ovn-acl-logging/0.log"
Sep 18 14:28:57.344061 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:28:57.344030    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:29:10.361884 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:10.361845    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:29:21.442079 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:21.442049    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:29:21.721891 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:21.721792    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b"]
Sep 18 14:29:21.722130 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:21.722117    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58292587-7ef6-4a85-a3b2-fede8ba8ca4c" containerName="istio-proxy"
Sep 18 14:29:21.722130 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:21.722131    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="58292587-7ef6-4a85-a3b2-fede8ba8ca4c" containerName="istio-proxy"
Sep 18 14:29:21.722282 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:21.722145    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4978be46-34db-43ce-87e2-a9c3bd6df751" containerName="istio-proxy"
Sep 18 14:29:21.722282 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:21.722151    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4978be46-34db-43ce-87e2-a9c3bd6df751" containerName="istio-proxy"
Sep 18 14:29:21.722282 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:21.722244    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4978be46-34db-43ce-87e2-a9c3bd6df751" containerName="istio-proxy"
Sep 18 14:29:21.724810 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:21.724789    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b"
Sep 18 14:29:21.727053 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:21.727033    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-b58948-openshift-default-dockercfg-ddn7d\""
Sep 18 14:29:21.739361 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:21.739339    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b"]
Sep 18 14:29:21.750247 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:21.750225    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:29:21.829100 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:21.829066    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18378eb4-074c-43fa-ab1b-9c623aba59cc-istio-data\") pod \"e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b\" (UID: \"18378eb4-074c-43fa-ab1b-9c623aba59cc\") " pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b"
Sep 18 14:29:21.829312 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:21.829160    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18378eb4-074c-43fa-ab1b-9c623aba59cc-istio-envoy\") pod \"e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b\" (UID: \"18378eb4-074c-43fa-ab1b-9c623aba59cc\") " pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b"
Sep 18 14:29:21.829312 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:21.829228    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18378eb4-074c-43fa-ab1b-9c623aba59cc-workload-socket\") pod \"e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b\" (UID: \"18378eb4-074c-43fa-ab1b-9c623aba59cc\") " pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b"
Sep 18 14:29:21.829312 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:21.829250    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18378eb4-074c-43fa-ab1b-9c623aba59cc-istio-podinfo\") pod \"e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b\" (UID: \"18378eb4-074c-43fa-ab1b-9c623aba59cc\") " pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b"
Sep 18 14:29:21.829312 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:21.829280    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18378eb4-074c-43fa-ab1b-9c623aba59cc-credential-socket\") pod \"e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b\" (UID: \"18378eb4-074c-43fa-ab1b-9c623aba59cc\") " pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b"
Sep 18 14:29:21.829539 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:21.829371    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18378eb4-074c-43fa-ab1b-9c623aba59cc-istiod-ca-cert\") pod \"e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b\" (UID: \"18378eb4-074c-43fa-ab1b-9c623aba59cc\") " pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b"
Sep 18 14:29:21.829539 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:21.829422    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18378eb4-074c-43fa-ab1b-9c623aba59cc-workload-certs\") pod \"e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b\" (UID: \"18378eb4-074c-43fa-ab1b-9c623aba59cc\") " pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b"
Sep 18 14:29:21.829539 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:21.829466    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rbwc\" (UniqueName: \"kubernetes.io/projected/18378eb4-074c-43fa-ab1b-9c623aba59cc-kube-api-access-8rbwc\") pod \"e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b\" (UID: \"18378eb4-074c-43fa-ab1b-9c623aba59cc\") " pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b"
Sep 18 14:29:21.829703 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:21.829537    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18378eb4-074c-43fa-ab1b-9c623aba59cc-istio-token\") pod \"e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b\" (UID: \"18378eb4-074c-43fa-ab1b-9c623aba59cc\") " pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b"
Sep 18 14:29:21.930529 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:21.930500    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18378eb4-074c-43fa-ab1b-9c623aba59cc-istio-token\") pod \"e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b\" (UID: \"18378eb4-074c-43fa-ab1b-9c623aba59cc\") " pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b"
Sep 18 14:29:21.930701 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:21.930550    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18378eb4-074c-43fa-ab1b-9c623aba59cc-istio-data\") pod \"e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b\" (UID: \"18378eb4-074c-43fa-ab1b-9c623aba59cc\") " pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b"
Sep 18 14:29:21.930701 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:21.930575    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18378eb4-074c-43fa-ab1b-9c623aba59cc-istio-envoy\") pod \"e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b\" (UID: \"18378eb4-074c-43fa-ab1b-9c623aba59cc\") " pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b"
Sep 18 14:29:21.930701 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:21.930592    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18378eb4-074c-43fa-ab1b-9c623aba59cc-workload-socket\") pod \"e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b\" (UID: \"18378eb4-074c-43fa-ab1b-9c623aba59cc\") " pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b"
Sep 18 14:29:21.930701 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:21.930609    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18378eb4-074c-43fa-ab1b-9c623aba59cc-istio-podinfo\") pod \"e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b\" (UID: \"18378eb4-074c-43fa-ab1b-9c623aba59cc\") " pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b"
Sep 18 14:29:21.930701 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:21.930626    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18378eb4-074c-43fa-ab1b-9c623aba59cc-credential-socket\") pod \"e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b\" (UID: \"18378eb4-074c-43fa-ab1b-9c623aba59cc\") " pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b"
Sep 18 14:29:21.930701 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:21.930648    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18378eb4-074c-43fa-ab1b-9c623aba59cc-istiod-ca-cert\") pod \"e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b\" (UID: \"18378eb4-074c-43fa-ab1b-9c623aba59cc\") " pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b"
Sep 18 14:29:21.930701 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:21.930666    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18378eb4-074c-43fa-ab1b-9c623aba59cc-workload-certs\") pod \"e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b\" (UID: \"18378eb4-074c-43fa-ab1b-9c623aba59cc\") " pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b"
Sep 18 14:29:21.930701 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:21.930686    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rbwc\" (UniqueName: \"kubernetes.io/projected/18378eb4-074c-43fa-ab1b-9c623aba59cc-kube-api-access-8rbwc\") pod \"e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b\" (UID: \"18378eb4-074c-43fa-ab1b-9c623aba59cc\") " pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b"
Sep 18 14:29:21.931137 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:21.931112    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18378eb4-074c-43fa-ab1b-9c623aba59cc-istio-data\") pod \"e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b\" (UID: \"18378eb4-074c-43fa-ab1b-9c623aba59cc\") " pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b"
Sep 18 14:29:21.931285 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:21.931261    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18378eb4-074c-43fa-ab1b-9c623aba59cc-workload-socket\") pod \"e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b\" (UID: \"18378eb4-074c-43fa-ab1b-9c623aba59cc\") " pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b"
Sep 18 14:29:21.931353 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:21.931283    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18378eb4-074c-43fa-ab1b-9c623aba59cc-workload-certs\") pod \"e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b\" (UID: \"18378eb4-074c-43fa-ab1b-9c623aba59cc\") " pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b"
Sep 18 14:29:21.931353 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:21.931314    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18378eb4-074c-43fa-ab1b-9c623aba59cc-credential-socket\") pod \"e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b\" (UID: \"18378eb4-074c-43fa-ab1b-9c623aba59cc\") " pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b"
Sep 18 14:29:21.931528 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:21.931504    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18378eb4-074c-43fa-ab1b-9c623aba59cc-istiod-ca-cert\") pod \"e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b\" (UID: \"18378eb4-074c-43fa-ab1b-9c623aba59cc\") " pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b"
Sep 18 14:29:21.933107 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:21.933084    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18378eb4-074c-43fa-ab1b-9c623aba59cc-istio-envoy\") pod \"e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b\" (UID: \"18378eb4-074c-43fa-ab1b-9c623aba59cc\") " pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b"
Sep 18 14:29:21.933264 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:21.933246    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18378eb4-074c-43fa-ab1b-9c623aba59cc-istio-podinfo\") pod \"e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b\" (UID: \"18378eb4-074c-43fa-ab1b-9c623aba59cc\") " pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b"
Sep 18 14:29:21.938572 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:21.938546    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18378eb4-074c-43fa-ab1b-9c623aba59cc-istio-token\") pod \"e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b\" (UID: \"18378eb4-074c-43fa-ab1b-9c623aba59cc\") " pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b"
Sep 18 14:29:21.938731 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:21.938714    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rbwc\" (UniqueName: \"kubernetes.io/projected/18378eb4-074c-43fa-ab1b-9c623aba59cc-kube-api-access-8rbwc\") pod \"e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b\" (UID: \"18378eb4-074c-43fa-ab1b-9c623aba59cc\") " pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b"
Sep 18 14:29:22.035085 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:22.035033    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b"
Sep 18 14:29:22.245731 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:22.245691    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:29:22.313891 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:22.313867    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b"]
Sep 18 14:29:22.315741 ip-10-0-143-120 kubenswrapper[2579]: W0918 14:29:22.315716    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18378eb4_074c_43fa_ab1b_9c623aba59cc.slice/crio-aa3fb345f25c60858efcee05099e2a6f55a41889d1c8e5b4215bc715f128d64c WatchSource:0}: Error finding container aa3fb345f25c60858efcee05099e2a6f55a41889d1c8e5b4215bc715f128d64c: Status 404 returned error can't find the container with id aa3fb345f25c60858efcee05099e2a6f55a41889d1c8e5b4215bc715f128d64c
Sep 18 14:29:22.318118 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:22.318086    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:29:22.318227 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:22.318150    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:29:22.318227 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:22.318195    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:29:22.508321 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:22.508276    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b" event={"ID":"18378eb4-074c-43fa-ab1b-9c623aba59cc","Type":"ContainerStarted","Data":"ad4f24c40df0327db4d1db2b98fce33ae73bc267b929db4cdb449c0eefb90db8"}
Sep 18 14:29:22.508727 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:22.508372    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b" event={"ID":"18378eb4-074c-43fa-ab1b-9c623aba59cc","Type":"ContainerStarted","Data":"aa3fb345f25c60858efcee05099e2a6f55a41889d1c8e5b4215bc715f128d64c"}
Sep 18 14:29:22.527905 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:22.527858    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b" podStartSLOduration=1.527844736 podStartE2EDuration="1.527844736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:29:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:29:22.525912986 +0000 UTC m=+1834.414628921" watchObservedRunningTime="2026-09-18 14:29:22.527844736 +0000 UTC m=+1834.416560667"
Sep 18 14:29:23.035417 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:23.035378    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b"
Sep 18 14:29:23.040589 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:23.040561    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b"
Sep 18 14:29:23.513504 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:23.513469    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b"
Sep 18 14:29:23.514491 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:23.514472    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b"
Sep 18 14:29:23.716477 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:23.716443    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:29:24.403928 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:24.403899    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:29:24.498400 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:24.498371    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:29:25.421910 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:25.421877    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:29:26.149596 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:26.149567    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:29:26.861310 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:26.861281    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:29:27.515978 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:27.515911    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b58948-66b77b9c48-4qr2t"]
Sep 18 14:29:27.519496 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:27.519477    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b58948-66b77b9c48-4qr2t"
Sep 18 14:29:27.521910 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:27.521889    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-b58948-serving-cert\""
Sep 18 14:29:27.531791 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:27.531750    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b58948-66b77b9c48-4qr2t"]
Sep 18 14:29:27.552738 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:27.552709    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:29:27.576647 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:27.576621    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9528a331-3d95-4c50-a6a8-ed6fb201a6fa-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-b58948-66b77b9c48-4qr2t\" (UID: \"9528a331-3d95-4c50-a6a8-ed6fb201a6fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b58948-66b77b9c48-4qr2t"
Sep 18 14:29:27.576743 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:27.576671    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9528a331-3d95-4c50-a6a8-ed6fb201a6fa-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-b58948-66b77b9c48-4qr2t\" (UID: \"9528a331-3d95-4c50-a6a8-ed6fb201a6fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b58948-66b77b9c48-4qr2t"
Sep 18 14:29:27.576743 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:27.576699    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9528a331-3d95-4c50-a6a8-ed6fb201a6fa-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-b58948-66b77b9c48-4qr2t\" (UID: \"9528a331-3d95-4c50-a6a8-ed6fb201a6fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b58948-66b77b9c48-4qr2t"
Sep 18 14:29:27.576841 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:27.576772    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gffq9\" (UniqueName: \"kubernetes.io/projected/9528a331-3d95-4c50-a6a8-ed6fb201a6fa-kube-api-access-gffq9\") pod \"maas-api-e2e-ipp-cleanup-b58948-66b77b9c48-4qr2t\" (UID: \"9528a331-3d95-4c50-a6a8-ed6fb201a6fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b58948-66b77b9c48-4qr2t"
Sep 18 14:29:27.654289 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:27.654260    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:29:27.677418 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:27.677389    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9528a331-3d95-4c50-a6a8-ed6fb201a6fa-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-b58948-66b77b9c48-4qr2t\" (UID: \"9528a331-3d95-4c50-a6a8-ed6fb201a6fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b58948-66b77b9c48-4qr2t"
Sep 18 14:29:27.677550 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:27.677424    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9528a331-3d95-4c50-a6a8-ed6fb201a6fa-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-b58948-66b77b9c48-4qr2t\" (UID: \"9528a331-3d95-4c50-a6a8-ed6fb201a6fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b58948-66b77b9c48-4qr2t"
Sep 18 14:29:27.677550 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:27.677451    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gffq9\" (UniqueName: \"kubernetes.io/projected/9528a331-3d95-4c50-a6a8-ed6fb201a6fa-kube-api-access-gffq9\") pod \"maas-api-e2e-ipp-cleanup-b58948-66b77b9c48-4qr2t\" (UID: \"9528a331-3d95-4c50-a6a8-ed6fb201a6fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b58948-66b77b9c48-4qr2t"
Sep 18 14:29:27.677550 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:27.677518    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9528a331-3d95-4c50-a6a8-ed6fb201a6fa-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-b58948-66b77b9c48-4qr2t\" (UID: \"9528a331-3d95-4c50-a6a8-ed6fb201a6fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b58948-66b77b9c48-4qr2t"
Sep 18 14:29:27.677726 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:29:27.677598    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b58948-serving-cert: secret "maas-api-e2e-ipp-cleanup-b58948-serving-cert" not found
Sep 18 14:29:27.677726 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:29:27.677656    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9528a331-3d95-4c50-a6a8-ed6fb201a6fa-maas-api-tls podName:9528a331-3d95-4c50-a6a8-ed6fb201a6fa nodeName:}" failed. No retries permitted until 2026-09-18 14:29:28.177642762 +0000 UTC m=+1840.066358671 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9528a331-3d95-4c50-a6a8-ed6fb201a6fa-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-b58948-66b77b9c48-4qr2t" (UID: "9528a331-3d95-4c50-a6a8-ed6fb201a6fa") : secret "maas-api-e2e-ipp-cleanup-b58948-serving-cert" not found
Sep 18 14:29:27.678141 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:27.678087    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9528a331-3d95-4c50-a6a8-ed6fb201a6fa-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-b58948-66b77b9c48-4qr2t\" (UID: \"9528a331-3d95-4c50-a6a8-ed6fb201a6fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b58948-66b77b9c48-4qr2t"
Sep 18 14:29:27.679887 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:27.679870    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9528a331-3d95-4c50-a6a8-ed6fb201a6fa-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-b58948-66b77b9c48-4qr2t\" (UID: \"9528a331-3d95-4c50-a6a8-ed6fb201a6fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b58948-66b77b9c48-4qr2t"
Sep 18 14:29:27.685642 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:27.685624    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gffq9\" (UniqueName: \"kubernetes.io/projected/9528a331-3d95-4c50-a6a8-ed6fb201a6fa-kube-api-access-gffq9\") pod \"maas-api-e2e-ipp-cleanup-b58948-66b77b9c48-4qr2t\" (UID: \"9528a331-3d95-4c50-a6a8-ed6fb201a6fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b58948-66b77b9c48-4qr2t"
Sep 18 14:29:27.999456 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:27.999426    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:29:28.098081 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:28.098044    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:29:28.181942 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:28.181907    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9528a331-3d95-4c50-a6a8-ed6fb201a6fa-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-b58948-66b77b9c48-4qr2t\" (UID: \"9528a331-3d95-4c50-a6a8-ed6fb201a6fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b58948-66b77b9c48-4qr2t"
Sep 18 14:29:28.184348 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:28.184322    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9528a331-3d95-4c50-a6a8-ed6fb201a6fa-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-b58948-66b77b9c48-4qr2t\" (UID: \"9528a331-3d95-4c50-a6a8-ed6fb201a6fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b58948-66b77b9c48-4qr2t"
Sep 18 14:29:28.195923 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:28.195901    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b58948-79fbf4f7d6-96bkn"]
Sep 18 14:29:28.199689 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:28.199672    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b58948-79fbf4f7d6-96bkn"
Sep 18 14:29:28.201830 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:28.201812    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-b58948\""
Sep 18 14:29:28.201903 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:28.201812    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-b58948-dockercfg-4x6c9\""
Sep 18 14:29:28.209064 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:28.209044    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b58948-79fbf4f7d6-96bkn"]
Sep 18 14:29:28.282620 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:28.282547    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65v2l\" (UniqueName: \"kubernetes.io/projected/ffaf57ae-d1d0-4b44-9599-257371c559d0-kube-api-access-65v2l\") pod \"payload-pre-processing-e2e-ipp-cleanup-b58948-79fbf4f7d6-96bkn\" (UID: \"ffaf57ae-d1d0-4b44-9599-257371c559d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b58948-79fbf4f7d6-96bkn"
Sep 18 14:29:28.282620 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:28.282592    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffaf57ae-d1d0-4b44-9599-257371c559d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-b58948-79fbf4f7d6-96bkn\" (UID: \"ffaf57ae-d1d0-4b44-9599-257371c559d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b58948-79fbf4f7d6-96bkn"
Sep 18 14:29:28.383876 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:28.383849    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65v2l\" (UniqueName: \"kubernetes.io/projected/ffaf57ae-d1d0-4b44-9599-257371c559d0-kube-api-access-65v2l\") pod \"payload-pre-processing-e2e-ipp-cleanup-b58948-79fbf4f7d6-96bkn\" (UID: \"ffaf57ae-d1d0-4b44-9599-257371c559d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b58948-79fbf4f7d6-96bkn"
Sep 18 14:29:28.384014 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:28.383890    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffaf57ae-d1d0-4b44-9599-257371c559d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-b58948-79fbf4f7d6-96bkn\" (UID: \"ffaf57ae-d1d0-4b44-9599-257371c559d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b58948-79fbf4f7d6-96bkn"
Sep 18 14:29:28.384611 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:28.384592    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffaf57ae-d1d0-4b44-9599-257371c559d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-b58948-79fbf4f7d6-96bkn\" (UID: \"ffaf57ae-d1d0-4b44-9599-257371c559d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b58948-79fbf4f7d6-96bkn"
Sep 18 14:29:28.391819 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:28.391790    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65v2l\" (UniqueName: \"kubernetes.io/projected/ffaf57ae-d1d0-4b44-9599-257371c559d0-kube-api-access-65v2l\") pod \"payload-pre-processing-e2e-ipp-cleanup-b58948-79fbf4f7d6-96bkn\" (UID: \"ffaf57ae-d1d0-4b44-9599-257371c559d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b58948-79fbf4f7d6-96bkn"
Sep 18 14:29:28.432771 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:28.432745    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b58948-66b77b9c48-4qr2t"
Sep 18 14:29:28.510857 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:28.510831    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b58948-79fbf4f7d6-96bkn"
Sep 18 14:29:28.703113 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:28.703091    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b58948-66b77b9c48-4qr2t"]
Sep 18 14:29:28.705269 ip-10-0-143-120 kubenswrapper[2579]: W0918 14:29:28.705243    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9528a331_3d95_4c50_a6a8_ed6fb201a6fa.slice/crio-1c6b753db44679eee2c0ac134ef9f797d89a98d3f4a85165645489be5cd5df21 WatchSource:0}: Error finding container 1c6b753db44679eee2c0ac134ef9f797d89a98d3f4a85165645489be5cd5df21: Status 404 returned error can't find the container with id 1c6b753db44679eee2c0ac134ef9f797d89a98d3f4a85165645489be5cd5df21
Sep 18 14:29:28.706548 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:28.706528    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 14:29:28.783220 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:28.783201    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b58948-79fbf4f7d6-96bkn"]
Sep 18 14:29:28.784980 ip-10-0-143-120 kubenswrapper[2579]: W0918 14:29:28.784957    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffaf57ae_d1d0_4b44_9599_257371c559d0.slice/crio-c14db5e5a3c48fefe553e8a29249c0194a3245112212cd3a20543960aa637ad6 WatchSource:0}: Error finding container c14db5e5a3c48fefe553e8a29249c0194a3245112212cd3a20543960aa637ad6: Status 404 returned error can't find the container with id c14db5e5a3c48fefe553e8a29249c0194a3245112212cd3a20543960aa637ad6
Sep 18 14:29:29.545119 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:29.545036    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b58948-66b77b9c48-4qr2t" event={"ID":"9528a331-3d95-4c50-a6a8-ed6fb201a6fa","Type":"ContainerStarted","Data":"84bc03a9cfa793d2272a7c14175aa069a95f3d5cda9a05c7395b797c083479d7"}
Sep 18 14:29:29.545119 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:29.545084    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b58948-66b77b9c48-4qr2t" event={"ID":"9528a331-3d95-4c50-a6a8-ed6fb201a6fa","Type":"ContainerStarted","Data":"1c6b753db44679eee2c0ac134ef9f797d89a98d3f4a85165645489be5cd5df21"}
Sep 18 14:29:29.545119 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:29.545109    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b58948-66b77b9c48-4qr2t"
Sep 18 14:29:29.546660 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:29.546633    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b58948-79fbf4f7d6-96bkn" event={"ID":"ffaf57ae-d1d0-4b44-9599-257371c559d0","Type":"ContainerStarted","Data":"a50ead205ca09ebca3fffadc2b1982b4ad7ce686b3d1d93d9a055123865945eb"}
Sep 18 14:29:29.546790 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:29.546680    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b58948-79fbf4f7d6-96bkn" event={"ID":"ffaf57ae-d1d0-4b44-9599-257371c559d0","Type":"ContainerStarted","Data":"c14db5e5a3c48fefe553e8a29249c0194a3245112212cd3a20543960aa637ad6"}
Sep 18 14:29:29.546790 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:29.546706    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b58948-79fbf4f7d6-96bkn"
Sep 18 14:29:29.564746 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:29.564699    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b58948-66b77b9c48-4qr2t" podStartSLOduration=2.106409712 podStartE2EDuration="2.56468437s" totalImagesPullingTime="458.274658ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:29:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:29:29.562914744 +0000 UTC m=+1841.451630702" watchObservedRunningTime="2026-09-18 14:29:29.56468437 +0000 UTC m=+1841.453400300"
Sep 18 14:29:29.580450 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:29.580402    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b58948-79fbf4f7d6-96bkn" podStartSLOduration=1.580382981 podStartE2EDuration="1.580382981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:29:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:29:29.579600278 +0000 UTC m=+1841.468316209" watchObservedRunningTime="2026-09-18 14:29:29.580382981 +0000 UTC m=+1841.469099135"
Sep 18 14:29:35.563522 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:35.563492    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b58948-66b77b9c48-4qr2t"
Sep 18 14:29:40.552909 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:40.552874    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b58948-79fbf4f7d6-96bkn"
Sep 18 14:29:41.491889 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:41.491847    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:29:43.619236 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:43.619129    2579 generic.go:356] "Generic (PLEG): container finished" podID="18378eb4-074c-43fa-ab1b-9c623aba59cc" containerID="ad4f24c40df0327db4d1db2b98fce33ae73bc267b929db4cdb449c0eefb90db8" exitCode=0
Sep 18 14:29:43.619236 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:43.619209    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b" event={"ID":"18378eb4-074c-43fa-ab1b-9c623aba59cc","Type":"ContainerDied","Data":"ad4f24c40df0327db4d1db2b98fce33ae73bc267b929db4cdb449c0eefb90db8"}
Sep 18 14:29:43.619837 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:43.619714    2579 scope.go:117] "RemoveContainer" containerID="ad4f24c40df0327db4d1db2b98fce33ae73bc267b929db4cdb449c0eefb90db8"
Sep 18 14:29:43.620944 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:43.620915    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:29:43.621007 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:43.620985    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:29:43.621049 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:43.621018    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 14:29:44.626043 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:44.626009    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b" event={"ID":"18378eb4-074c-43fa-ab1b-9c623aba59cc","Type":"ContainerStarted","Data":"6a06eac2755b5e40218718084cd71360b2626cb5b802d92475c8b089b0649f83"}
Sep 18 14:29:45.035470 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:45.035381    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b"
Sep 18 14:29:45.035621 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:45.035600    2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 18 14:29:45.035673 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:45.035658    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b" podUID="18378eb4-074c-43fa-ab1b-9c623aba59cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 18 14:29:46.035777 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:46.035745    2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 18 14:29:46.036138 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:46.035792    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b" podUID="18378eb4-074c-43fa-ab1b-9c623aba59cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 18 14:29:47.036137 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:47.036101    2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 18 14:29:47.036541 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:47.036153    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b" podUID="18378eb4-074c-43fa-ab1b-9c623aba59cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 18 14:29:48.035488 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:48.035450    2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 18 14:29:48.035684 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:48.035509    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b" podUID="18378eb4-074c-43fa-ab1b-9c623aba59cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 18 14:29:49.036091 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:49.036058    2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 18 14:29:49.036497 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:49.036151    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b" podUID="18378eb4-074c-43fa-ab1b-9c623aba59cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 18 14:29:49.078817 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:49.078754    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:29:49.342638 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:49.342603    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:29:49.454843 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:49.454799    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:29:49.997108 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:49.997078    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:29:50.040151 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:50.040124    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b"
Sep 18 14:29:50.040592 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:50.040422    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b"
Sep 18 14:29:50.041360 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:50.041344    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b"
Sep 18 14:29:50.329572 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:50.329540    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:29:51.151745 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:51.151713    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:29:51.839829 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:51.839797    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:29:52.728439 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:52.728405    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:29:53.424900 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:53.424859    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:29:54.118135 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:54.118102    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:29:54.926770 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:54.926739    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:29:55.718462 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:55.718429    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:29:56.246433 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:56.246391    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b58948-66b77b9c48-4qr2t"]
Sep 18 14:29:56.246665 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:56.246635    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b58948-66b77b9c48-4qr2t" podUID="9528a331-3d95-4c50-a6a8-ed6fb201a6fa" containerName="maas-api" containerID="cri-o://84bc03a9cfa793d2272a7c14175aa069a95f3d5cda9a05c7395b797c083479d7" gracePeriod=30
Sep 18 14:29:56.277456 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:56.277431    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:29:56.320424 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:56.320390    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b58948-79fbf4f7d6-96bkn"]
Sep 18 14:29:56.320817 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:56.320641    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b58948-79fbf4f7d6-96bkn" podUID="ffaf57ae-d1d0-4b44-9599-257371c559d0" containerName="payload-pre-processing" containerID="cri-o://a50ead205ca09ebca3fffadc2b1982b4ad7ce686b3d1d93d9a055123865945eb" gracePeriod=30
Sep 18 14:29:56.374753 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:56.374724    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:29:56.476937 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:56.476880    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:29:56.552458 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:56.552430    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:29:56.577280 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:56.577255    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b58948-66b77b9c48-4qr2t"
Sep 18 14:29:56.620451 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:56.620425    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b58948-79fbf4f7d6-96bkn"
Sep 18 14:29:56.649620 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:56.649598    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9528a331-3d95-4c50-a6a8-ed6fb201a6fa-maas-api-tls\") pod \"9528a331-3d95-4c50-a6a8-ed6fb201a6fa\" (UID: \"9528a331-3d95-4c50-a6a8-ed6fb201a6fa\") "
Sep 18 14:29:56.649756 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:56.649698    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9528a331-3d95-4c50-a6a8-ed6fb201a6fa-metrics-certs\") pod \"9528a331-3d95-4c50-a6a8-ed6fb201a6fa\" (UID: \"9528a331-3d95-4c50-a6a8-ed6fb201a6fa\") "
Sep 18 14:29:56.649756 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:56.649726    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9528a331-3d95-4c50-a6a8-ed6fb201a6fa-openshift-service-ca\") pod \"9528a331-3d95-4c50-a6a8-ed6fb201a6fa\" (UID: \"9528a331-3d95-4c50-a6a8-ed6fb201a6fa\") "
Sep 18 14:29:56.649937 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:56.649902    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gffq9\" (UniqueName: \"kubernetes.io/projected/9528a331-3d95-4c50-a6a8-ed6fb201a6fa-kube-api-access-gffq9\") pod \"9528a331-3d95-4c50-a6a8-ed6fb201a6fa\" (UID: \"9528a331-3d95-4c50-a6a8-ed6fb201a6fa\") "
Sep 18 14:29:56.650079 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:56.650055    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9528a331-3d95-4c50-a6a8-ed6fb201a6fa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9528a331-3d95-4c50-a6a8-ed6fb201a6fa" (UID: "9528a331-3d95-4c50-a6a8-ed6fb201a6fa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:29:56.650319 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:56.650296    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9528a331-3d95-4c50-a6a8-ed6fb201a6fa-openshift-service-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:29:56.652098 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:56.652071    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9528a331-3d95-4c50-a6a8-ed6fb201a6fa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9528a331-3d95-4c50-a6a8-ed6fb201a6fa" (UID: "9528a331-3d95-4c50-a6a8-ed6fb201a6fa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:29:56.652242 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:56.652115    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9528a331-3d95-4c50-a6a8-ed6fb201a6fa-kube-api-access-gffq9" (OuterVolumeSpecName: "kube-api-access-gffq9") pod "9528a331-3d95-4c50-a6a8-ed6fb201a6fa" (UID: "9528a331-3d95-4c50-a6a8-ed6fb201a6fa"). InnerVolumeSpecName "kube-api-access-gffq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:29:56.652242 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:56.652128    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9528a331-3d95-4c50-a6a8-ed6fb201a6fa-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9528a331-3d95-4c50-a6a8-ed6fb201a6fa" (UID: "9528a331-3d95-4c50-a6a8-ed6fb201a6fa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:29:56.689906 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:56.689882    2579 generic.go:356] "Generic (PLEG): container finished" podID="9528a331-3d95-4c50-a6a8-ed6fb201a6fa" containerID="84bc03a9cfa793d2272a7c14175aa069a95f3d5cda9a05c7395b797c083479d7" exitCode=0
Sep 18 14:29:56.689994 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:56.689924    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b58948-66b77b9c48-4qr2t"
Sep 18 14:29:56.689994 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:56.689967    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b58948-66b77b9c48-4qr2t" event={"ID":"9528a331-3d95-4c50-a6a8-ed6fb201a6fa","Type":"ContainerDied","Data":"84bc03a9cfa793d2272a7c14175aa069a95f3d5cda9a05c7395b797c083479d7"}
Sep 18 14:29:56.690099 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:56.690020    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b58948-66b77b9c48-4qr2t" event={"ID":"9528a331-3d95-4c50-a6a8-ed6fb201a6fa","Type":"ContainerDied","Data":"1c6b753db44679eee2c0ac134ef9f797d89a98d3f4a85165645489be5cd5df21"}
Sep 18 14:29:56.690099 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:56.690042    2579 scope.go:117] "RemoveContainer" containerID="84bc03a9cfa793d2272a7c14175aa069a95f3d5cda9a05c7395b797c083479d7"
Sep 18 14:29:56.691340 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:56.691319    2579 generic.go:356] "Generic (PLEG): container finished" podID="ffaf57ae-d1d0-4b44-9599-257371c559d0" containerID="a50ead205ca09ebca3fffadc2b1982b4ad7ce686b3d1d93d9a055123865945eb" exitCode=0
Sep 18 14:29:56.691454 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:56.691372    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b58948-79fbf4f7d6-96bkn"
Sep 18 14:29:56.691454 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:56.691410    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b58948-79fbf4f7d6-96bkn" event={"ID":"ffaf57ae-d1d0-4b44-9599-257371c559d0","Type":"ContainerDied","Data":"a50ead205ca09ebca3fffadc2b1982b4ad7ce686b3d1d93d9a055123865945eb"}
Sep 18 14:29:56.691626 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:56.691479    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b58948-79fbf4f7d6-96bkn" event={"ID":"ffaf57ae-d1d0-4b44-9599-257371c559d0","Type":"ContainerDied","Data":"c14db5e5a3c48fefe553e8a29249c0194a3245112212cd3a20543960aa637ad6"}
Sep 18 14:29:56.698848 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:56.698831    2579 scope.go:117] "RemoveContainer" containerID="84bc03a9cfa793d2272a7c14175aa069a95f3d5cda9a05c7395b797c083479d7"
Sep 18 14:29:56.699093 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:29:56.699076    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84bc03a9cfa793d2272a7c14175aa069a95f3d5cda9a05c7395b797c083479d7\": container with ID starting with 84bc03a9cfa793d2272a7c14175aa069a95f3d5cda9a05c7395b797c083479d7 not found: ID does not exist" containerID="84bc03a9cfa793d2272a7c14175aa069a95f3d5cda9a05c7395b797c083479d7"
Sep 18 14:29:56.699142 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:56.699101    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84bc03a9cfa793d2272a7c14175aa069a95f3d5cda9a05c7395b797c083479d7"} err="failed to get container status \"84bc03a9cfa793d2272a7c14175aa069a95f3d5cda9a05c7395b797c083479d7\": rpc error: code = NotFound desc = could not find container \"84bc03a9cfa793d2272a7c14175aa069a95f3d5cda9a05c7395b797c083479d7\": container with ID starting with 84bc03a9cfa793d2272a7c14175aa069a95f3d5cda9a05c7395b797c083479d7 not found: ID does not exist"
Sep 18 14:29:56.699142 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:56.699117    2579 scope.go:117] "RemoveContainer" containerID="a50ead205ca09ebca3fffadc2b1982b4ad7ce686b3d1d93d9a055123865945eb"
Sep 18 14:29:56.706199 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:56.706181    2579 scope.go:117] "RemoveContainer" containerID="a50ead205ca09ebca3fffadc2b1982b4ad7ce686b3d1d93d9a055123865945eb"
Sep 18 14:29:56.706453 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:29:56.706435    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a50ead205ca09ebca3fffadc2b1982b4ad7ce686b3d1d93d9a055123865945eb\": container with ID starting with a50ead205ca09ebca3fffadc2b1982b4ad7ce686b3d1d93d9a055123865945eb not found: ID does not exist" containerID="a50ead205ca09ebca3fffadc2b1982b4ad7ce686b3d1d93d9a055123865945eb"
Sep 18 14:29:56.706510 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:56.706458    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a50ead205ca09ebca3fffadc2b1982b4ad7ce686b3d1d93d9a055123865945eb"} err="failed to get container status \"a50ead205ca09ebca3fffadc2b1982b4ad7ce686b3d1d93d9a055123865945eb\": rpc error: code = NotFound desc = could not find container \"a50ead205ca09ebca3fffadc2b1982b4ad7ce686b3d1d93d9a055123865945eb\": container with ID starting with a50ead205ca09ebca3fffadc2b1982b4ad7ce686b3d1d93d9a055123865945eb not found: ID does not exist"
Sep 18 14:29:56.711430 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:56.711408    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b58948-66b77b9c48-4qr2t"]
Sep 18 14:29:56.714551 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:56.714532    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b58948-66b77b9c48-4qr2t"]
Sep 18 14:29:56.751073 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:56.751050    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffaf57ae-d1d0-4b44-9599-257371c559d0-plugins-config-volume\") pod \"ffaf57ae-d1d0-4b44-9599-257371c559d0\" (UID: \"ffaf57ae-d1d0-4b44-9599-257371c559d0\") "
Sep 18 14:29:56.751480 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:56.751090    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65v2l\" (UniqueName: \"kubernetes.io/projected/ffaf57ae-d1d0-4b44-9599-257371c559d0-kube-api-access-65v2l\") pod \"ffaf57ae-d1d0-4b44-9599-257371c559d0\" (UID: \"ffaf57ae-d1d0-4b44-9599-257371c559d0\") "
Sep 18 14:29:56.751480 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:56.751418    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9528a331-3d95-4c50-a6a8-ed6fb201a6fa-maas-api-tls\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:29:56.751480 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:56.751429    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9528a331-3d95-4c50-a6a8-ed6fb201a6fa-metrics-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:29:56.751480 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:56.751439    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gffq9\" (UniqueName: \"kubernetes.io/projected/9528a331-3d95-4c50-a6a8-ed6fb201a6fa-kube-api-access-gffq9\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:29:56.751671 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:56.751527    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffaf57ae-d1d0-4b44-9599-257371c559d0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ffaf57ae-d1d0-4b44-9599-257371c559d0" (UID: "ffaf57ae-d1d0-4b44-9599-257371c559d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:29:56.753062 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:56.753043    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffaf57ae-d1d0-4b44-9599-257371c559d0-kube-api-access-65v2l" (OuterVolumeSpecName: "kube-api-access-65v2l") pod "ffaf57ae-d1d0-4b44-9599-257371c559d0" (UID: "ffaf57ae-d1d0-4b44-9599-257371c559d0"). InnerVolumeSpecName "kube-api-access-65v2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:29:56.851793 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:56.851770    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffaf57ae-d1d0-4b44-9599-257371c559d0-plugins-config-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:29:56.851793 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:56.851790    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65v2l\" (UniqueName: \"kubernetes.io/projected/ffaf57ae-d1d0-4b44-9599-257371c559d0-kube-api-access-65v2l\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:29:56.862827 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:56.862803    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:29:56.962461 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:56.962435    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:29:57.026713 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:57.026682    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b58948-79fbf4f7d6-96bkn"]
Sep 18 14:29:57.030472 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:57.030444    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b58948-79fbf4f7d6-96bkn"]
Sep 18 14:29:57.660527 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:57.660496    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:29:58.034987 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:58.034911    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9528a331-3d95-4c50-a6a8-ed6fb201a6fa" path="/var/lib/kubelet/pods/9528a331-3d95-4c50-a6a8-ed6fb201a6fa/volumes"
Sep 18 14:29:58.035363 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:58.035287    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffaf57ae-d1d0-4b44-9599-257371c559d0" path="/var/lib/kubelet/pods/ffaf57ae-d1d0-4b44-9599-257371c559d0/volumes"
Sep 18 14:29:58.562239 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:58.562202    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:29:59.503743 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:29:59.503702    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:30:00.232413 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:00.232382    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ae54af-29829030-zsv8l"]
Sep 18 14:30:00.232755 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:00.232742    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9528a331-3d95-4c50-a6a8-ed6fb201a6fa" containerName="maas-api"
Sep 18 14:30:00.232801 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:00.232757    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9528a331-3d95-4c50-a6a8-ed6fb201a6fa" containerName="maas-api"
Sep 18 14:30:00.232801 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:00.232773    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffaf57ae-d1d0-4b44-9599-257371c559d0" containerName="payload-pre-processing"
Sep 18 14:30:00.232801 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:00.232782    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffaf57ae-d1d0-4b44-9599-257371c559d0" containerName="payload-pre-processing"
Sep 18 14:30:00.232900 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:00.232844    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffaf57ae-d1d0-4b44-9599-257371c559d0" containerName="payload-pre-processing"
Sep 18 14:30:00.232900 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:00.232855    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9528a331-3d95-4c50-a6a8-ed6fb201a6fa" containerName="maas-api"
Sep 18 14:30:00.235426 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:00.235404    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-980661-29829030-t8hdh"]
Sep 18 14:30:00.235571 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:00.235546    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ae54af-29829030-zsv8l"
Sep 18 14:30:00.238061 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:00.238041    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4fn9\""
Sep 18 14:30:00.246798 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:00.246779    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-980661-29829030-t8hdh"
Sep 18 14:30:00.247662 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:00.247648    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ae54af-29829030-zsv8l"
Sep 18 14:30:00.257608 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:00.257225    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-980661-29829030-t8hdh"
Sep 18 14:30:00.262417 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:00.262392    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ae54af-29829030-zsv8l"]
Sep 18 14:30:00.262544 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:00.262446    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-980661-29829030-t8hdh"]
Sep 18 14:30:00.550250 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:00.550218    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ae54af-29829030-zsv8l"]
Sep 18 14:30:00.550951 ip-10-0-143-120 kubenswrapper[2579]: W0918 14:30:00.550915    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0bdf88b_cc5a_4bd2_b929_6b49403b09d6.slice/crio-b58fd3f048106d1eb3d80c950b32683380fa41ccd8f0717759bee75235cecdd3 WatchSource:0}: Error finding container b58fd3f048106d1eb3d80c950b32683380fa41ccd8f0717759bee75235cecdd3: Status 404 returned error can't find the container with id b58fd3f048106d1eb3d80c950b32683380fa41ccd8f0717759bee75235cecdd3
Sep 18 14:30:00.551885 ip-10-0-143-120 kubenswrapper[2579]: W0918 14:30:00.551859    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cb744c0_bb08_4656_999b_a5957595de6c.slice/crio-412164edf5ea5979d3b1afcab615a7da57ed6b4c16031fa22d15b82f0dedc017 WatchSource:0}: Error finding container 412164edf5ea5979d3b1afcab615a7da57ed6b4c16031fa22d15b82f0dedc017: Status 404 returned error can't find the container with id 412164edf5ea5979d3b1afcab615a7da57ed6b4c16031fa22d15b82f0dedc017
Sep 18 14:30:00.552131 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:00.552071    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-980661-29829030-t8hdh"]
Sep 18 14:30:00.712028 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:00.712000    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ae54af-29829030-zsv8l" event={"ID":"f0bdf88b-cc5a-4bd2-b929-6b49403b09d6","Type":"ContainerStarted","Data":"b58fd3f048106d1eb3d80c950b32683380fa41ccd8f0717759bee75235cecdd3"}
Sep 18 14:30:00.713152 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:00.713129    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-980661-29829030-t8hdh" event={"ID":"8cb744c0-bb08-4656-999b-a5957595de6c","Type":"ContainerStarted","Data":"412164edf5ea5979d3b1afcab615a7da57ed6b4c16031fa22d15b82f0dedc017"}
Sep 18 14:30:00.744103 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:00.744081    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:30:03.164866 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:03.164836    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:30:03.347167 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:03.347087    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:30:03.471785 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:03.471754    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b"]
Sep 18 14:30:03.472007 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:03.471982    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b" podUID="18378eb4-074c-43fa-ab1b-9c623aba59cc" containerName="istio-proxy" containerID="cri-o://6a06eac2755b5e40218718084cd71360b2626cb5b802d92475c8b089b0649f83" gracePeriod=30
Sep 18 14:30:03.730620 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:03.730529    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ae54af-29829030-zsv8l" event={"ID":"f0bdf88b-cc5a-4bd2-b929-6b49403b09d6","Type":"ContainerStarted","Data":"4de0e1a6410693625c206767d157594ea1648a7187daa755bd4368412679b53b"}
Sep 18 14:30:03.732026 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:03.731997    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-980661-29829030-t8hdh" event={"ID":"8cb744c0-bb08-4656-999b-a5957595de6c","Type":"ContainerStarted","Data":"668048be0ff704928eff6542a96751ebc5ae70fe0a486a2d39de9b8a51ccdd03"}
Sep 18 14:30:03.748473 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:03.748436    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ae54af-29829030-zsv8l" podStartSLOduration=1.473637381 podStartE2EDuration="3.748423558s" totalImagesPullingTime="2.274786177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:30:03.747725238 +0000 UTC m=+1875.636441170" watchObservedRunningTime="2026-09-18 14:30:03.748423558 +0000 UTC m=+1875.637139533"
Sep 18 14:30:03.749580 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:03.749561    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:30:03.763776 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:03.763743    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-980661-29829030-t8hdh" podStartSLOduration=1.485147686 podStartE2EDuration="3.763730944s" totalImagesPullingTime="2.278583258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:30:03.762413422 +0000 UTC m=+1875.651129354" watchObservedRunningTime="2026-09-18 14:30:03.763730944 +0000 UTC m=+1875.652446935"
Sep 18 14:30:04.737971 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:04.737938    2579 generic.go:356] "Generic (PLEG): container finished" podID="f0bdf88b-cc5a-4bd2-b929-6b49403b09d6" containerID="4de0e1a6410693625c206767d157594ea1648a7187daa755bd4368412679b53b" exitCode=0
Sep 18 14:30:04.738394 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:04.738018    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ae54af-29829030-zsv8l" event={"ID":"f0bdf88b-cc5a-4bd2-b929-6b49403b09d6","Type":"ContainerDied","Data":"4de0e1a6410693625c206767d157594ea1648a7187daa755bd4368412679b53b"}
Sep 18 14:30:04.739567 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:04.739542    2579 generic.go:356] "Generic (PLEG): container finished" podID="8cb744c0-bb08-4656-999b-a5957595de6c" containerID="668048be0ff704928eff6542a96751ebc5ae70fe0a486a2d39de9b8a51ccdd03" exitCode=0
Sep 18 14:30:04.739707 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:04.739620    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-980661-29829030-t8hdh" event={"ID":"8cb744c0-bb08-4656-999b-a5957595de6c","Type":"ContainerDied","Data":"668048be0ff704928eff6542a96751ebc5ae70fe0a486a2d39de9b8a51ccdd03"}
Sep 18 14:30:05.043240 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:05.043134    2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 14:30:05.043240 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:05.043204    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b" podUID="18378eb4-074c-43fa-ab1b-9c623aba59cc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 14:30:05.895436 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:05.895404    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-980661-29829030-t8hdh"
Sep 18 14:30:05.898058 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:05.898041    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ae54af-29829030-zsv8l"
Sep 18 14:30:06.750362 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:06.750331    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ae54af-29829030-zsv8l"
Sep 18 14:30:06.750362 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:06.750344    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ae54af-29829030-zsv8l" event={"ID":"f0bdf88b-cc5a-4bd2-b929-6b49403b09d6","Type":"ContainerDied","Data":"b58fd3f048106d1eb3d80c950b32683380fa41ccd8f0717759bee75235cecdd3"}
Sep 18 14:30:06.750596 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:06.750390    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b58fd3f048106d1eb3d80c950b32683380fa41ccd8f0717759bee75235cecdd3"
Sep 18 14:30:06.751643 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:06.751618    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-980661-29829030-t8hdh" event={"ID":"8cb744c0-bb08-4656-999b-a5957595de6c","Type":"ContainerDied","Data":"412164edf5ea5979d3b1afcab615a7da57ed6b4c16031fa22d15b82f0dedc017"}
Sep 18 14:30:06.751643 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:06.751635    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-980661-29829030-t8hdh"
Sep 18 14:30:06.751801 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:06.751662    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="412164edf5ea5979d3b1afcab615a7da57ed6b4c16031fa22d15b82f0dedc017"
Sep 18 14:30:07.298052 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:07.298025    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:30:08.726820 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:08.726798    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b"
Sep 18 14:30:08.765370 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:08.765294    2579 generic.go:356] "Generic (PLEG): container finished" podID="18378eb4-074c-43fa-ab1b-9c623aba59cc" containerID="6a06eac2755b5e40218718084cd71360b2626cb5b802d92475c8b089b0649f83" exitCode=0
Sep 18 14:30:08.765370 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:08.765347    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b"
Sep 18 14:30:08.765559 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:08.765380    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b" event={"ID":"18378eb4-074c-43fa-ab1b-9c623aba59cc","Type":"ContainerDied","Data":"6a06eac2755b5e40218718084cd71360b2626cb5b802d92475c8b089b0649f83"}
Sep 18 14:30:08.765559 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:08.765443    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b" event={"ID":"18378eb4-074c-43fa-ab1b-9c623aba59cc","Type":"ContainerDied","Data":"aa3fb345f25c60858efcee05099e2a6f55a41889d1c8e5b4215bc715f128d64c"}
Sep 18 14:30:08.765559 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:08.765465    2579 scope.go:117] "RemoveContainer" containerID="6a06eac2755b5e40218718084cd71360b2626cb5b802d92475c8b089b0649f83"
Sep 18 14:30:08.774161 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:08.774145    2579 scope.go:117] "RemoveContainer" containerID="ad4f24c40df0327db4d1db2b98fce33ae73bc267b929db4cdb449c0eefb90db8"
Sep 18 14:30:08.782275 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:08.782257    2579 scope.go:117] "RemoveContainer" containerID="6a06eac2755b5e40218718084cd71360b2626cb5b802d92475c8b089b0649f83"
Sep 18 14:30:08.782545 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:30:08.782517    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a06eac2755b5e40218718084cd71360b2626cb5b802d92475c8b089b0649f83\": container with ID starting with 6a06eac2755b5e40218718084cd71360b2626cb5b802d92475c8b089b0649f83 not found: ID does not exist" containerID="6a06eac2755b5e40218718084cd71360b2626cb5b802d92475c8b089b0649f83"
Sep 18 14:30:08.782590 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:08.782554    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a06eac2755b5e40218718084cd71360b2626cb5b802d92475c8b089b0649f83"} err="failed to get container status \"6a06eac2755b5e40218718084cd71360b2626cb5b802d92475c8b089b0649f83\": rpc error: code = NotFound desc = could not find container \"6a06eac2755b5e40218718084cd71360b2626cb5b802d92475c8b089b0649f83\": container with ID starting with 6a06eac2755b5e40218718084cd71360b2626cb5b802d92475c8b089b0649f83 not found: ID does not exist"
Sep 18 14:30:08.782590 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:08.782571    2579 scope.go:117] "RemoveContainer" containerID="ad4f24c40df0327db4d1db2b98fce33ae73bc267b929db4cdb449c0eefb90db8"
Sep 18 14:30:08.782832 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:30:08.782814    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad4f24c40df0327db4d1db2b98fce33ae73bc267b929db4cdb449c0eefb90db8\": container with ID starting with ad4f24c40df0327db4d1db2b98fce33ae73bc267b929db4cdb449c0eefb90db8 not found: ID does not exist" containerID="ad4f24c40df0327db4d1db2b98fce33ae73bc267b929db4cdb449c0eefb90db8"
Sep 18 14:30:08.782891 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:08.782836    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad4f24c40df0327db4d1db2b98fce33ae73bc267b929db4cdb449c0eefb90db8"} err="failed to get container status \"ad4f24c40df0327db4d1db2b98fce33ae73bc267b929db4cdb449c0eefb90db8\": rpc error: code = NotFound desc = could not find container \"ad4f24c40df0327db4d1db2b98fce33ae73bc267b929db4cdb449c0eefb90db8\": container with ID starting with ad4f24c40df0327db4d1db2b98fce33ae73bc267b929db4cdb449c0eefb90db8 not found: ID does not exist"
Sep 18 14:30:08.865038 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:08.865011    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18378eb4-074c-43fa-ab1b-9c623aba59cc-workload-socket\") pod \"18378eb4-074c-43fa-ab1b-9c623aba59cc\" (UID: \"18378eb4-074c-43fa-ab1b-9c623aba59cc\") "
Sep 18 14:30:08.865142 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:08.865049    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8rbwc\" (UniqueName: \"kubernetes.io/projected/18378eb4-074c-43fa-ab1b-9c623aba59cc-kube-api-access-8rbwc\") pod \"18378eb4-074c-43fa-ab1b-9c623aba59cc\" (UID: \"18378eb4-074c-43fa-ab1b-9c623aba59cc\") "
Sep 18 14:30:08.865142 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:08.865083    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18378eb4-074c-43fa-ab1b-9c623aba59cc-istio-envoy\") pod \"18378eb4-074c-43fa-ab1b-9c623aba59cc\" (UID: \"18378eb4-074c-43fa-ab1b-9c623aba59cc\") "
Sep 18 14:30:08.865142 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:08.865098    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18378eb4-074c-43fa-ab1b-9c623aba59cc-istio-token\") pod \"18378eb4-074c-43fa-ab1b-9c623aba59cc\" (UID: \"18378eb4-074c-43fa-ab1b-9c623aba59cc\") "
Sep 18 14:30:08.865142 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:08.865139    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18378eb4-074c-43fa-ab1b-9c623aba59cc-istiod-ca-cert\") pod \"18378eb4-074c-43fa-ab1b-9c623aba59cc\" (UID: \"18378eb4-074c-43fa-ab1b-9c623aba59cc\") "
Sep 18 14:30:08.865378 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:08.865166    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18378eb4-074c-43fa-ab1b-9c623aba59cc-istio-podinfo\") pod \"18378eb4-074c-43fa-ab1b-9c623aba59cc\" (UID: \"18378eb4-074c-43fa-ab1b-9c623aba59cc\") "
Sep 18 14:30:08.865378 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:08.865211    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18378eb4-074c-43fa-ab1b-9c623aba59cc-credential-socket\") pod \"18378eb4-074c-43fa-ab1b-9c623aba59cc\" (UID: \"18378eb4-074c-43fa-ab1b-9c623aba59cc\") "
Sep 18 14:30:08.865378 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:08.865259    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18378eb4-074c-43fa-ab1b-9c623aba59cc-workload-certs\") pod \"18378eb4-074c-43fa-ab1b-9c623aba59cc\" (UID: \"18378eb4-074c-43fa-ab1b-9c623aba59cc\") "
Sep 18 14:30:08.865378 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:08.865263    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18378eb4-074c-43fa-ab1b-9c623aba59cc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "18378eb4-074c-43fa-ab1b-9c623aba59cc" (UID: "18378eb4-074c-43fa-ab1b-9c623aba59cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:30:08.865378 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:08.865319    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18378eb4-074c-43fa-ab1b-9c623aba59cc-istio-data\") pod \"18378eb4-074c-43fa-ab1b-9c623aba59cc\" (UID: \"18378eb4-074c-43fa-ab1b-9c623aba59cc\") "
Sep 18 14:30:08.865646 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:08.865486    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18378eb4-074c-43fa-ab1b-9c623aba59cc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "18378eb4-074c-43fa-ab1b-9c623aba59cc" (UID: "18378eb4-074c-43fa-ab1b-9c623aba59cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:30:08.865646 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:08.865629    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18378eb4-074c-43fa-ab1b-9c623aba59cc-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:30:08.865646 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:08.865634    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18378eb4-074c-43fa-ab1b-9c623aba59cc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "18378eb4-074c-43fa-ab1b-9c623aba59cc" (UID: "18378eb4-074c-43fa-ab1b-9c623aba59cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:30:08.865812 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:08.865648    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18378eb4-074c-43fa-ab1b-9c623aba59cc-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:30:08.865812 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:08.865660    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18378eb4-074c-43fa-ab1b-9c623aba59cc-istio-data" (OuterVolumeSpecName: "istio-data") pod "18378eb4-074c-43fa-ab1b-9c623aba59cc" (UID: "18378eb4-074c-43fa-ab1b-9c623aba59cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:30:08.865921 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:08.865818    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18378eb4-074c-43fa-ab1b-9c623aba59cc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "18378eb4-074c-43fa-ab1b-9c623aba59cc" (UID: "18378eb4-074c-43fa-ab1b-9c623aba59cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:30:08.867435 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:08.867406    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18378eb4-074c-43fa-ab1b-9c623aba59cc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "18378eb4-074c-43fa-ab1b-9c623aba59cc" (UID: "18378eb4-074c-43fa-ab1b-9c623aba59cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:30:08.867557 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:08.867530    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18378eb4-074c-43fa-ab1b-9c623aba59cc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "18378eb4-074c-43fa-ab1b-9c623aba59cc" (UID: "18378eb4-074c-43fa-ab1b-9c623aba59cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 14:30:08.867716 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:08.867691    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18378eb4-074c-43fa-ab1b-9c623aba59cc-istio-token" (OuterVolumeSpecName: "istio-token") pod "18378eb4-074c-43fa-ab1b-9c623aba59cc" (UID: "18378eb4-074c-43fa-ab1b-9c623aba59cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:30:08.867813 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:08.867720    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18378eb4-074c-43fa-ab1b-9c623aba59cc-kube-api-access-8rbwc" (OuterVolumeSpecName: "kube-api-access-8rbwc") pod "18378eb4-074c-43fa-ab1b-9c623aba59cc" (UID: "18378eb4-074c-43fa-ab1b-9c623aba59cc"). InnerVolumeSpecName "kube-api-access-8rbwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:30:08.881396 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:08.881372    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:30:08.966804 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:08.966781    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18378eb4-074c-43fa-ab1b-9c623aba59cc-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:30:08.966804 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:08.966799    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18378eb4-074c-43fa-ab1b-9c623aba59cc-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:30:08.966956 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:08.966808    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18378eb4-074c-43fa-ab1b-9c623aba59cc-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:30:08.966956 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:08.966816    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18378eb4-074c-43fa-ab1b-9c623aba59cc-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:30:08.966956 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:08.966845    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rbwc\" (UniqueName: \"kubernetes.io/projected/18378eb4-074c-43fa-ab1b-9c623aba59cc-kube-api-access-8rbwc\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:30:08.966956 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:08.966854    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18378eb4-074c-43fa-ab1b-9c623aba59cc-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:30:08.966956 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:08.966863    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18378eb4-074c-43fa-ab1b-9c623aba59cc-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:30:08.975253 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:08.975234    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:30:09.090350 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:09.090318    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b"]
Sep 18 14:30:09.095681 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:09.095660    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-b58948-openshift-default-7c696896cb-ttw2b"]
Sep 18 14:30:09.842689 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:09.842654    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:30:10.035156 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:10.035122    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18378eb4-074c-43fa-ab1b-9c623aba59cc" path="/var/lib/kubelet/pods/18378eb4-074c-43fa-ab1b-9c623aba59cc/volumes"
Sep 18 14:30:10.840191 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:10.840093    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:30:11.702536 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:11.702502    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:30:22.642774 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:22.642742    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:30:22.714341 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:22.714311    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:30:22.780422 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:22.780356    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:30:22.853197 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:22.853135    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:30:22.916298 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:22.916217    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:30:23.214060 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:23.213941    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:30:23.327300 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:23.327277    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:30:24.036213 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:24.036164    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:30:24.828337 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:24.828301    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:30:25.524419 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:25.524385    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:30:28.290790 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:28.290739    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:30:51.199224 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:30:51.199163    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:31:04.909700 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:04.909669    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:31:05.914822 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:05.914786    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:31:11.492625 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:11.492587    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:31:26.270997 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:26.270956    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:31:26.357518 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:26.357483    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:31:26.677541 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:26.677511    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:31:27.306489 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:27.306459    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:31:28.480939 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:28.480912    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:31:29.316538 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:29.316505    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:31:30.542879 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:30.542854    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:31:31.872447 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:31.872406    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:31:37.787117 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:37.787024    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:31:37.851212 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:37.851164    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7ed186-6596bd9ff8-jd7tp"]
Sep 18 14:31:37.851447 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:37.851397    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7ed186-6596bd9ff8-jd7tp" podUID="46f30e53-2dbf-41f9-abd4-82701d9594fa" containerName="payload-pre-processing" containerID="cri-o://76a14ca066398422b508bcd74083982061bd6672fb1e863aff0fcd9d9e563942" gracePeriod=30
Sep 18 14:31:37.925858 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:37.925831    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:31:38.042637 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:38.042567    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:31:38.098963 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:38.098939    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7ed186-6596bd9ff8-jd7tp"
Sep 18 14:31:38.183447 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:38.183422    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46f30e53-2dbf-41f9-abd4-82701d9594fa-plugins-config-volume\") pod \"46f30e53-2dbf-41f9-abd4-82701d9594fa\" (UID: \"46f30e53-2dbf-41f9-abd4-82701d9594fa\") "
Sep 18 14:31:38.183590 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:38.183460    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kqk9\" (UniqueName: \"kubernetes.io/projected/46f30e53-2dbf-41f9-abd4-82701d9594fa-kube-api-access-9kqk9\") pod \"46f30e53-2dbf-41f9-abd4-82701d9594fa\" (UID: \"46f30e53-2dbf-41f9-abd4-82701d9594fa\") "
Sep 18 14:31:38.183831 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:38.183799    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46f30e53-2dbf-41f9-abd4-82701d9594fa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "46f30e53-2dbf-41f9-abd4-82701d9594fa" (UID: "46f30e53-2dbf-41f9-abd4-82701d9594fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:31:38.185479 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:38.185455    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46f30e53-2dbf-41f9-abd4-82701d9594fa-kube-api-access-9kqk9" (OuterVolumeSpecName: "kube-api-access-9kqk9") pod "46f30e53-2dbf-41f9-abd4-82701d9594fa" (UID: "46f30e53-2dbf-41f9-abd4-82701d9594fa"). InnerVolumeSpecName "kube-api-access-9kqk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:31:38.198706 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:38.198682    2579 generic.go:356] "Generic (PLEG): container finished" podID="46f30e53-2dbf-41f9-abd4-82701d9594fa" containerID="76a14ca066398422b508bcd74083982061bd6672fb1e863aff0fcd9d9e563942" exitCode=0
Sep 18 14:31:38.198823 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:38.198728    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7ed186-6596bd9ff8-jd7tp"
Sep 18 14:31:38.198823 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:38.198734    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7ed186-6596bd9ff8-jd7tp" event={"ID":"46f30e53-2dbf-41f9-abd4-82701d9594fa","Type":"ContainerDied","Data":"76a14ca066398422b508bcd74083982061bd6672fb1e863aff0fcd9d9e563942"}
Sep 18 14:31:38.198823 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:38.198783    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7ed186-6596bd9ff8-jd7tp" event={"ID":"46f30e53-2dbf-41f9-abd4-82701d9594fa","Type":"ContainerDied","Data":"98aa76b93f754ffcbaddbf050306d6126b751b368d08bfce6d19aec067b6fd59"}
Sep 18 14:31:38.198823 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:38.198804    2579 scope.go:117] "RemoveContainer" containerID="76a14ca066398422b508bcd74083982061bd6672fb1e863aff0fcd9d9e563942"
Sep 18 14:31:38.209081 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:38.208980    2579 scope.go:117] "RemoveContainer" containerID="76a14ca066398422b508bcd74083982061bd6672fb1e863aff0fcd9d9e563942"
Sep 18 14:31:38.209628 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:31:38.209594    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76a14ca066398422b508bcd74083982061bd6672fb1e863aff0fcd9d9e563942\": container with ID starting with 76a14ca066398422b508bcd74083982061bd6672fb1e863aff0fcd9d9e563942 not found: ID does not exist" containerID="76a14ca066398422b508bcd74083982061bd6672fb1e863aff0fcd9d9e563942"
Sep 18 14:31:38.209738 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:38.209629    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76a14ca066398422b508bcd74083982061bd6672fb1e863aff0fcd9d9e563942"} err="failed to get container status \"76a14ca066398422b508bcd74083982061bd6672fb1e863aff0fcd9d9e563942\": rpc error: code = NotFound desc = could not find container \"76a14ca066398422b508bcd74083982061bd6672fb1e863aff0fcd9d9e563942\": container with ID starting with 76a14ca066398422b508bcd74083982061bd6672fb1e863aff0fcd9d9e563942 not found: ID does not exist"
Sep 18 14:31:38.210538 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:38.210518    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:31:38.241530 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:38.241503    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7ed186-6596bd9ff8-jd7tp"]
Sep 18 14:31:38.244567 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:38.244539    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7ed186-6596bd9ff8-jd7tp"]
Sep 18 14:31:38.284608 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:38.284583    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kqk9\" (UniqueName: \"kubernetes.io/projected/46f30e53-2dbf-41f9-abd4-82701d9594fa-kube-api-access-9kqk9\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:31:38.284608 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:38.284606    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46f30e53-2dbf-41f9-abd4-82701d9594fa-plugins-config-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:31:38.313890 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:38.313866    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:31:38.639608 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:38.639548    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:31:38.770224 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:38.770197    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:31:39.796221 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:39.796166    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:31:40.035837 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:40.035806    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46f30e53-2dbf-41f9-abd4-82701d9594fa" path="/var/lib/kubelet/pods/46f30e53-2dbf-41f9-abd4-82701d9594fa/volumes"
Sep 18 14:31:40.908334 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:40.908283    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:31:45.237109 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:45.237079    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:31:45.312983 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:45.312955    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:31:45.640917 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:45.640887    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:31:46.256654 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:46.256623    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:31:47.314020 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:47.313983    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:31:48.155212 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:48.155158    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:31:49.077519 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:49.077485    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:31:56.509891 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:56.509856    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ae54af-29829030-zsv8l"]
Sep 18 14:31:56.510766 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:56.510745    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:31:56.517596 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:56.517572    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ae54af-29829030-zsv8l"]
Sep 18 14:31:56.560816 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:56.560784    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ae54af-6b864dd554-7kk9c"]
Sep 18 14:31:56.561300 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:56.561266    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ae54af-6b864dd554-7kk9c" podUID="6b11577e-8cc1-47f2-b262-6c2043dcafb0" containerName="payload-pre-processing" containerID="cri-o://328d484f2ba3760706836a4a5044146d2295bf3c8ea185a6c71cee9d3c99f83e" gracePeriod=30
Sep 18 14:31:56.611133 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:56.611105    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:31:56.710878 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:56.710842    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:31:56.820258 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:56.820229    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ae54af-6b864dd554-7kk9c"
Sep 18 14:31:56.824566 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:56.824532    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:31:56.915270 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:56.915237    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xbgns\" (UniqueName: \"kubernetes.io/projected/6b11577e-8cc1-47f2-b262-6c2043dcafb0-kube-api-access-xbgns\") pod \"6b11577e-8cc1-47f2-b262-6c2043dcafb0\" (UID: \"6b11577e-8cc1-47f2-b262-6c2043dcafb0\") "
Sep 18 14:31:56.915442 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:56.915277    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b11577e-8cc1-47f2-b262-6c2043dcafb0-plugins-config-volume\") pod \"6b11577e-8cc1-47f2-b262-6c2043dcafb0\" (UID: \"6b11577e-8cc1-47f2-b262-6c2043dcafb0\") "
Sep 18 14:31:56.915718 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:56.915694    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b11577e-8cc1-47f2-b262-6c2043dcafb0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6b11577e-8cc1-47f2-b262-6c2043dcafb0" (UID: "6b11577e-8cc1-47f2-b262-6c2043dcafb0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:31:56.917503 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:56.917483    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b11577e-8cc1-47f2-b262-6c2043dcafb0-kube-api-access-xbgns" (OuterVolumeSpecName: "kube-api-access-xbgns") pod "6b11577e-8cc1-47f2-b262-6c2043dcafb0" (UID: "6b11577e-8cc1-47f2-b262-6c2043dcafb0"). InnerVolumeSpecName "kube-api-access-xbgns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:31:56.919562 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:56.919538    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:31:57.015953 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:57.015902    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbgns\" (UniqueName: \"kubernetes.io/projected/6b11577e-8cc1-47f2-b262-6c2043dcafb0-kube-api-access-xbgns\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:31:57.015953 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:57.015945    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b11577e-8cc1-47f2-b262-6c2043dcafb0-plugins-config-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:31:57.238857 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:57.238765    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:31:57.295248 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:57.295215    2579 generic.go:356] "Generic (PLEG): container finished" podID="6b11577e-8cc1-47f2-b262-6c2043dcafb0" containerID="328d484f2ba3760706836a4a5044146d2295bf3c8ea185a6c71cee9d3c99f83e" exitCode=0
Sep 18 14:31:57.295448 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:57.295258    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ae54af-6b864dd554-7kk9c"
Sep 18 14:31:57.295448 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:57.295268    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ae54af-6b864dd554-7kk9c" event={"ID":"6b11577e-8cc1-47f2-b262-6c2043dcafb0","Type":"ContainerDied","Data":"328d484f2ba3760706836a4a5044146d2295bf3c8ea185a6c71cee9d3c99f83e"}
Sep 18 14:31:57.295448 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:57.295320    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ae54af-6b864dd554-7kk9c" event={"ID":"6b11577e-8cc1-47f2-b262-6c2043dcafb0","Type":"ContainerDied","Data":"290bb963ca206bd7b07cc2dbff323901ede29f19fc4fdfffa2429fb2085d865a"}
Sep 18 14:31:57.295448 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:57.295348    2579 scope.go:117] "RemoveContainer" containerID="328d484f2ba3760706836a4a5044146d2295bf3c8ea185a6c71cee9d3c99f83e"
Sep 18 14:31:57.303940 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:57.303919    2579 scope.go:117] "RemoveContainer" containerID="328d484f2ba3760706836a4a5044146d2295bf3c8ea185a6c71cee9d3c99f83e"
Sep 18 14:31:57.304274 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:31:57.304251    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"328d484f2ba3760706836a4a5044146d2295bf3c8ea185a6c71cee9d3c99f83e\": container with ID starting with 328d484f2ba3760706836a4a5044146d2295bf3c8ea185a6c71cee9d3c99f83e not found: ID does not exist" containerID="328d484f2ba3760706836a4a5044146d2295bf3c8ea185a6c71cee9d3c99f83e"
Sep 18 14:31:57.304345 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:57.304288    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"328d484f2ba3760706836a4a5044146d2295bf3c8ea185a6c71cee9d3c99f83e"} err="failed to get container status \"328d484f2ba3760706836a4a5044146d2295bf3c8ea185a6c71cee9d3c99f83e\": rpc error: code = NotFound desc = could not find container \"328d484f2ba3760706836a4a5044146d2295bf3c8ea185a6c71cee9d3c99f83e\": container with ID starting with 328d484f2ba3760706836a4a5044146d2295bf3c8ea185a6c71cee9d3c99f83e not found: ID does not exist"
Sep 18 14:31:57.327356 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:57.327324    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ae54af-6b864dd554-7kk9c"]
Sep 18 14:31:57.335210 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:57.335157    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ae54af-6b864dd554-7kk9c"]
Sep 18 14:31:57.342925 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:57.342871    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:31:58.035716 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:58.035688    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b11577e-8cc1-47f2-b262-6c2043dcafb0" path="/var/lib/kubelet/pods/6b11577e-8cc1-47f2-b262-6c2043dcafb0/volumes"
Sep 18 14:31:58.036081 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:58.035948    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0bdf88b-cc5a-4bd2-b929-6b49403b09d6" path="/var/lib/kubelet/pods/f0bdf88b-cc5a-4bd2-b929-6b49403b09d6/volumes"
Sep 18 14:31:58.238637 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:31:58.238609    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:32:03.828498 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:03.828465    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:32:03.929267 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:03.929238    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:32:04.068587 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:04.068557    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp"]
Sep 18 14:32:04.068774 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:04.068752    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp" podUID="b95b7806-9984-4565-a427-06f996c82677" containerName="istio-proxy" containerID="cri-o://c0fc8fe4b2e9498b02be786d7548807b0ae4f352c3c7ebc6e1b4f673617e5796" gracePeriod=30
Sep 18 14:32:04.094480 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:04.094409    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:32:05.420088 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:05.420057    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:32:06.513994 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:06.513958    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:32:09.317372 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:09.317348    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp"
Sep 18 14:32:09.353082 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:09.353017    2579 generic.go:356] "Generic (PLEG): container finished" podID="b95b7806-9984-4565-a427-06f996c82677" containerID="c0fc8fe4b2e9498b02be786d7548807b0ae4f352c3c7ebc6e1b4f673617e5796" exitCode=0
Sep 18 14:32:09.353082 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:09.353052    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp"
Sep 18 14:32:09.353295 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:09.353094    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp" event={"ID":"b95b7806-9984-4565-a427-06f996c82677","Type":"ContainerDied","Data":"c0fc8fe4b2e9498b02be786d7548807b0ae4f352c3c7ebc6e1b4f673617e5796"}
Sep 18 14:32:09.353295 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:09.353149    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp" event={"ID":"b95b7806-9984-4565-a427-06f996c82677","Type":"ContainerDied","Data":"7b1d2b5496430a54ed6f26d492e6b0d13b9553a9f2072942b1b8c448b262bd6a"}
Sep 18 14:32:09.353295 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:09.353197    2579 scope.go:117] "RemoveContainer" containerID="c0fc8fe4b2e9498b02be786d7548807b0ae4f352c3c7ebc6e1b4f673617e5796"
Sep 18 14:32:09.361206 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:09.361185    2579 scope.go:117] "RemoveContainer" containerID="32c4a0eff350c5c7a09ed6b225d23a25ea6c388bdc6427cada95f588b8e19d19"
Sep 18 14:32:09.368707 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:09.368690    2579 scope.go:117] "RemoveContainer" containerID="c0fc8fe4b2e9498b02be786d7548807b0ae4f352c3c7ebc6e1b4f673617e5796"
Sep 18 14:32:09.368948 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:32:09.368931    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0fc8fe4b2e9498b02be786d7548807b0ae4f352c3c7ebc6e1b4f673617e5796\": container with ID starting with c0fc8fe4b2e9498b02be786d7548807b0ae4f352c3c7ebc6e1b4f673617e5796 not found: ID does not exist" containerID="c0fc8fe4b2e9498b02be786d7548807b0ae4f352c3c7ebc6e1b4f673617e5796"
Sep 18 14:32:09.369004 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:09.368956    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0fc8fe4b2e9498b02be786d7548807b0ae4f352c3c7ebc6e1b4f673617e5796"} err="failed to get container status \"c0fc8fe4b2e9498b02be786d7548807b0ae4f352c3c7ebc6e1b4f673617e5796\": rpc error: code = NotFound desc = could not find container \"c0fc8fe4b2e9498b02be786d7548807b0ae4f352c3c7ebc6e1b4f673617e5796\": container with ID starting with c0fc8fe4b2e9498b02be786d7548807b0ae4f352c3c7ebc6e1b4f673617e5796 not found: ID does not exist"
Sep 18 14:32:09.369004 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:09.368970    2579 scope.go:117] "RemoveContainer" containerID="32c4a0eff350c5c7a09ed6b225d23a25ea6c388bdc6427cada95f588b8e19d19"
Sep 18 14:32:09.369243 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:32:09.369223    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32c4a0eff350c5c7a09ed6b225d23a25ea6c388bdc6427cada95f588b8e19d19\": container with ID starting with 32c4a0eff350c5c7a09ed6b225d23a25ea6c388bdc6427cada95f588b8e19d19 not found: ID does not exist" containerID="32c4a0eff350c5c7a09ed6b225d23a25ea6c388bdc6427cada95f588b8e19d19"
Sep 18 14:32:09.369300 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:09.369250    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32c4a0eff350c5c7a09ed6b225d23a25ea6c388bdc6427cada95f588b8e19d19"} err="failed to get container status \"32c4a0eff350c5c7a09ed6b225d23a25ea6c388bdc6427cada95f588b8e19d19\": rpc error: code = NotFound desc = could not find container \"32c4a0eff350c5c7a09ed6b225d23a25ea6c388bdc6427cada95f588b8e19d19\": container with ID starting with 32c4a0eff350c5c7a09ed6b225d23a25ea6c388bdc6427cada95f588b8e19d19 not found: ID does not exist"
Sep 18 14:32:09.415698 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:09.415677    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b95b7806-9984-4565-a427-06f996c82677-credential-socket\") pod \"b95b7806-9984-4565-a427-06f996c82677\" (UID: \"b95b7806-9984-4565-a427-06f996c82677\") "
Sep 18 14:32:09.415781 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:09.415742    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b95b7806-9984-4565-a427-06f996c82677-istiod-ca-cert\") pod \"b95b7806-9984-4565-a427-06f996c82677\" (UID: \"b95b7806-9984-4565-a427-06f996c82677\") "
Sep 18 14:32:09.415781 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:09.415763    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b95b7806-9984-4565-a427-06f996c82677-istio-token\") pod \"b95b7806-9984-4565-a427-06f996c82677\" (UID: \"b95b7806-9984-4565-a427-06f996c82677\") "
Sep 18 14:32:09.415887 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:09.415801    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b95b7806-9984-4565-a427-06f996c82677-istio-envoy\") pod \"b95b7806-9984-4565-a427-06f996c82677\" (UID: \"b95b7806-9984-4565-a427-06f996c82677\") "
Sep 18 14:32:09.415887 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:09.415853    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b95b7806-9984-4565-a427-06f996c82677-istio-data\") pod \"b95b7806-9984-4565-a427-06f996c82677\" (UID: \"b95b7806-9984-4565-a427-06f996c82677\") "
Sep 18 14:32:09.415990 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:09.415892    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b95b7806-9984-4565-a427-06f996c82677-workload-socket\") pod \"b95b7806-9984-4565-a427-06f996c82677\" (UID: \"b95b7806-9984-4565-a427-06f996c82677\") "
Sep 18 14:32:09.415990 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:09.415927    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b95b7806-9984-4565-a427-06f996c82677-istio-podinfo\") pod \"b95b7806-9984-4565-a427-06f996c82677\" (UID: \"b95b7806-9984-4565-a427-06f996c82677\") "
Sep 18 14:32:09.415990 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:09.415975    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b95b7806-9984-4565-a427-06f996c82677-workload-certs\") pod \"b95b7806-9984-4565-a427-06f996c82677\" (UID: \"b95b7806-9984-4565-a427-06f996c82677\") "
Sep 18 14:32:09.416145 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:09.415999    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vkhk7\" (UniqueName: \"kubernetes.io/projected/b95b7806-9984-4565-a427-06f996c82677-kube-api-access-vkhk7\") pod \"b95b7806-9984-4565-a427-06f996c82677\" (UID: \"b95b7806-9984-4565-a427-06f996c82677\") "
Sep 18 14:32:09.416145 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:09.416112    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b95b7806-9984-4565-a427-06f996c82677-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b95b7806-9984-4565-a427-06f996c82677" (UID: "b95b7806-9984-4565-a427-06f996c82677"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:32:09.416145 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:09.416126    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b95b7806-9984-4565-a427-06f996c82677-istio-data" (OuterVolumeSpecName: "istio-data") pod "b95b7806-9984-4565-a427-06f996c82677" (UID: "b95b7806-9984-4565-a427-06f996c82677"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:32:09.416340 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:09.416214    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b95b7806-9984-4565-a427-06f996c82677-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b95b7806-9984-4565-a427-06f996c82677" (UID: "b95b7806-9984-4565-a427-06f996c82677"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:32:09.416340 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:09.416223    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b95b7806-9984-4565-a427-06f996c82677-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b95b7806-9984-4565-a427-06f996c82677" (UID: "b95b7806-9984-4565-a427-06f996c82677"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:32:09.416340 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:09.416333    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b95b7806-9984-4565-a427-06f996c82677-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b95b7806-9984-4565-a427-06f996c82677" (UID: "b95b7806-9984-4565-a427-06f996c82677"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:32:09.416503 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:09.416398    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b95b7806-9984-4565-a427-06f996c82677-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:32:09.416503 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:09.416410    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b95b7806-9984-4565-a427-06f996c82677-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:32:09.416503 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:09.416424    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b95b7806-9984-4565-a427-06f996c82677-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:32:09.416503 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:09.416438    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b95b7806-9984-4565-a427-06f996c82677-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:32:09.416503 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:09.416451    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b95b7806-9984-4565-a427-06f996c82677-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:32:09.417942 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:09.417918    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b95b7806-9984-4565-a427-06f996c82677-istio-token" (OuterVolumeSpecName: "istio-token") pod "b95b7806-9984-4565-a427-06f996c82677" (UID: "b95b7806-9984-4565-a427-06f996c82677"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:32:09.418036 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:09.418012    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b95b7806-9984-4565-a427-06f996c82677-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b95b7806-9984-4565-a427-06f996c82677" (UID: "b95b7806-9984-4565-a427-06f996c82677"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 14:32:09.418096 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:09.418052    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b95b7806-9984-4565-a427-06f996c82677-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b95b7806-9984-4565-a427-06f996c82677" (UID: "b95b7806-9984-4565-a427-06f996c82677"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:32:09.418162 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:09.418147    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b95b7806-9984-4565-a427-06f996c82677-kube-api-access-vkhk7" (OuterVolumeSpecName: "kube-api-access-vkhk7") pod "b95b7806-9984-4565-a427-06f996c82677" (UID: "b95b7806-9984-4565-a427-06f996c82677"). InnerVolumeSpecName "kube-api-access-vkhk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:32:09.517471 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:09.517440    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b95b7806-9984-4565-a427-06f996c82677-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:32:09.517471 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:09.517466    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b95b7806-9984-4565-a427-06f996c82677-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:32:09.517600 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:09.517477    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b95b7806-9984-4565-a427-06f996c82677-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:32:09.517600 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:09.517489    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkhk7\" (UniqueName: \"kubernetes.io/projected/b95b7806-9984-4565-a427-06f996c82677-kube-api-access-vkhk7\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:32:09.678406 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:09.678386    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp"]
Sep 18 14:32:09.682403 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:09.682385    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ae54af-openshift-default-6c9677b66d-2t6wp"]
Sep 18 14:32:10.040102 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:10.040037    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b95b7806-9984-4565-a427-06f996c82677" path="/var/lib/kubelet/pods/b95b7806-9984-4565-a427-06f996c82677/volumes"
Sep 18 14:32:15.044040 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:15.044010    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-980661-29829030-t8hdh"]
Sep 18 14:32:15.048777 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:15.048753    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-980661-29829030-t8hdh"]
Sep 18 14:32:15.048941 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:15.048779    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:32:15.082735 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:15.082708    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-980661-77765769cc-rvgpd"]
Sep 18 14:32:15.082937 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:15.082915    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-980661-77765769cc-rvgpd" podUID="0e570098-8720-40dd-944b-28607af93d0a" containerName="payload-processing" containerID="cri-o://2ea5d5490e25d0fdec6e6e3a127cf52916389c1fa807b77d43d478bd36e025fa" gracePeriod=30
Sep 18 14:32:15.089011 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:15.088987    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-980661-68db9c5866-5hkmc"]
Sep 18 14:32:15.089218 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:15.089196    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-980661-68db9c5866-5hkmc" podUID="f65a88fe-acd7-4b42-808f-81f80c051a3e" containerName="payload-pre-processing" containerID="cri-o://b4cf55ef37eb597f8bb447ecf0e1a4033ba57140f8909bd97590f11944612f45" gracePeriod=30
Sep 18 14:32:15.133001 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:15.132974    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:32:15.213507 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:15.213483    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:32:15.325549 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:15.325519    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:32:15.361076 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:15.361052    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-980661-77765769cc-rvgpd"
Sep 18 14:32:15.364046 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:15.364026    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-980661-68db9c5866-5hkmc"
Sep 18 14:32:15.384816 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:15.384793    2579 generic.go:356] "Generic (PLEG): container finished" podID="f65a88fe-acd7-4b42-808f-81f80c051a3e" containerID="b4cf55ef37eb597f8bb447ecf0e1a4033ba57140f8909bd97590f11944612f45" exitCode=0
Sep 18 14:32:15.384816 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:15.384825    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-980661-68db9c5866-5hkmc"
Sep 18 14:32:15.384976 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:15.384901    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-980661-68db9c5866-5hkmc" event={"ID":"f65a88fe-acd7-4b42-808f-81f80c051a3e","Type":"ContainerDied","Data":"b4cf55ef37eb597f8bb447ecf0e1a4033ba57140f8909bd97590f11944612f45"}
Sep 18 14:32:15.384976 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:15.384939    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-980661-68db9c5866-5hkmc" event={"ID":"f65a88fe-acd7-4b42-808f-81f80c051a3e","Type":"ContainerDied","Data":"9f6350296ba54e2b4c9fc67811188a14ab60f2ae379ab9c3e20620f1a1da0546"}
Sep 18 14:32:15.384976 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:15.384958    2579 scope.go:117] "RemoveContainer" containerID="b4cf55ef37eb597f8bb447ecf0e1a4033ba57140f8909bd97590f11944612f45"
Sep 18 14:32:15.386102 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:15.386084    2579 generic.go:356] "Generic (PLEG): container finished" podID="0e570098-8720-40dd-944b-28607af93d0a" containerID="2ea5d5490e25d0fdec6e6e3a127cf52916389c1fa807b77d43d478bd36e025fa" exitCode=0
Sep 18 14:32:15.386188 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:15.386115    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-980661-77765769cc-rvgpd"
Sep 18 14:32:15.386188 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:15.386137    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-980661-77765769cc-rvgpd" event={"ID":"0e570098-8720-40dd-944b-28607af93d0a","Type":"ContainerDied","Data":"2ea5d5490e25d0fdec6e6e3a127cf52916389c1fa807b77d43d478bd36e025fa"}
Sep 18 14:32:15.386188 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:15.386183    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-980661-77765769cc-rvgpd" event={"ID":"0e570098-8720-40dd-944b-28607af93d0a","Type":"ContainerDied","Data":"db43d6fd5e46259678bde1281001fad7dc260c3cca3f269b6d79b7ad81aed77b"}
Sep 18 14:32:15.393269 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:15.393243    2579 scope.go:117] "RemoveContainer" containerID="b4cf55ef37eb597f8bb447ecf0e1a4033ba57140f8909bd97590f11944612f45"
Sep 18 14:32:15.393682 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:32:15.393653    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4cf55ef37eb597f8bb447ecf0e1a4033ba57140f8909bd97590f11944612f45\": container with ID starting with b4cf55ef37eb597f8bb447ecf0e1a4033ba57140f8909bd97590f11944612f45 not found: ID does not exist" containerID="b4cf55ef37eb597f8bb447ecf0e1a4033ba57140f8909bd97590f11944612f45"
Sep 18 14:32:15.393799 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:15.393694    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4cf55ef37eb597f8bb447ecf0e1a4033ba57140f8909bd97590f11944612f45"} err="failed to get container status \"b4cf55ef37eb597f8bb447ecf0e1a4033ba57140f8909bd97590f11944612f45\": rpc error: code = NotFound desc = could not find container \"b4cf55ef37eb597f8bb447ecf0e1a4033ba57140f8909bd97590f11944612f45\": container with ID starting with b4cf55ef37eb597f8bb447ecf0e1a4033ba57140f8909bd97590f11944612f45 not found: ID does not exist"
Sep 18 14:32:15.393799 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:15.393717    2579 scope.go:117] "RemoveContainer" containerID="2ea5d5490e25d0fdec6e6e3a127cf52916389c1fa807b77d43d478bd36e025fa"
Sep 18 14:32:15.401942 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:15.401914    2579 scope.go:117] "RemoveContainer" containerID="2ea5d5490e25d0fdec6e6e3a127cf52916389c1fa807b77d43d478bd36e025fa"
Sep 18 14:32:15.402194 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:32:15.402151    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ea5d5490e25d0fdec6e6e3a127cf52916389c1fa807b77d43d478bd36e025fa\": container with ID starting with 2ea5d5490e25d0fdec6e6e3a127cf52916389c1fa807b77d43d478bd36e025fa not found: ID does not exist" containerID="2ea5d5490e25d0fdec6e6e3a127cf52916389c1fa807b77d43d478bd36e025fa"
Sep 18 14:32:15.402298 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:15.402203    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ea5d5490e25d0fdec6e6e3a127cf52916389c1fa807b77d43d478bd36e025fa"} err="failed to get container status \"2ea5d5490e25d0fdec6e6e3a127cf52916389c1fa807b77d43d478bd36e025fa\": rpc error: code = NotFound desc = could not find container \"2ea5d5490e25d0fdec6e6e3a127cf52916389c1fa807b77d43d478bd36e025fa\": container with ID starting with 2ea5d5490e25d0fdec6e6e3a127cf52916389c1fa807b77d43d478bd36e025fa not found: ID does not exist"
Sep 18 14:32:15.458584 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:15.458550    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f65a88fe-acd7-4b42-808f-81f80c051a3e-plugins-config-volume\") pod \"f65a88fe-acd7-4b42-808f-81f80c051a3e\" (UID: \"f65a88fe-acd7-4b42-808f-81f80c051a3e\") "
Sep 18 14:32:15.458785 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:15.458607    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h6rcc\" (UniqueName: \"kubernetes.io/projected/f65a88fe-acd7-4b42-808f-81f80c051a3e-kube-api-access-h6rcc\") pod \"f65a88fe-acd7-4b42-808f-81f80c051a3e\" (UID: \"f65a88fe-acd7-4b42-808f-81f80c051a3e\") "
Sep 18 14:32:15.458785 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:15.458641    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e570098-8720-40dd-944b-28607af93d0a-plugins-config-volume\") pod \"0e570098-8720-40dd-944b-28607af93d0a\" (UID: \"0e570098-8720-40dd-944b-28607af93d0a\") "
Sep 18 14:32:15.458785 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:15.458710    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b4hh7\" (UniqueName: \"kubernetes.io/projected/0e570098-8720-40dd-944b-28607af93d0a-kube-api-access-b4hh7\") pod \"0e570098-8720-40dd-944b-28607af93d0a\" (UID: \"0e570098-8720-40dd-944b-28607af93d0a\") "
Sep 18 14:32:15.458988 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:15.458954    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f65a88fe-acd7-4b42-808f-81f80c051a3e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f65a88fe-acd7-4b42-808f-81f80c051a3e" (UID: "f65a88fe-acd7-4b42-808f-81f80c051a3e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:32:15.459453 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:15.459421    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e570098-8720-40dd-944b-28607af93d0a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0e570098-8720-40dd-944b-28607af93d0a" (UID: "0e570098-8720-40dd-944b-28607af93d0a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:32:15.461555 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:15.461525    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f65a88fe-acd7-4b42-808f-81f80c051a3e-kube-api-access-h6rcc" (OuterVolumeSpecName: "kube-api-access-h6rcc") pod "f65a88fe-acd7-4b42-808f-81f80c051a3e" (UID: "f65a88fe-acd7-4b42-808f-81f80c051a3e"). InnerVolumeSpecName "kube-api-access-h6rcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:32:15.461691 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:15.461580    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e570098-8720-40dd-944b-28607af93d0a-kube-api-access-b4hh7" (OuterVolumeSpecName: "kube-api-access-b4hh7") pod "0e570098-8720-40dd-944b-28607af93d0a" (UID: "0e570098-8720-40dd-944b-28607af93d0a"). InnerVolumeSpecName "kube-api-access-b4hh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:32:15.559843 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:15.559820    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4hh7\" (UniqueName: \"kubernetes.io/projected/0e570098-8720-40dd-944b-28607af93d0a-kube-api-access-b4hh7\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:32:15.559843 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:15.559843    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f65a88fe-acd7-4b42-808f-81f80c051a3e-plugins-config-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:32:15.559979 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:15.559856    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6rcc\" (UniqueName: \"kubernetes.io/projected/f65a88fe-acd7-4b42-808f-81f80c051a3e-kube-api-access-h6rcc\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:32:15.559979 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:15.559866    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e570098-8720-40dd-944b-28607af93d0a-plugins-config-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:32:15.702253 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:15.702232    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:32:15.716907 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:15.716882    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-980661-68db9c5866-5hkmc"]
Sep 18 14:32:15.727365 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:15.727346    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-980661-68db9c5866-5hkmc"]
Sep 18 14:32:15.737676 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:15.737655    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-980661-77765769cc-rvgpd"]
Sep 18 14:32:15.744559 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:15.744540    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-980661-77765769cc-rvgpd"]
Sep 18 14:32:15.838556 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:15.838535    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:32:16.034573 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:16.034510    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e570098-8720-40dd-944b-28607af93d0a" path="/var/lib/kubelet/pods/0e570098-8720-40dd-944b-28607af93d0a/volumes"
Sep 18 14:32:16.034787 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:16.034774    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cb744c0-bb08-4656-999b-a5957595de6c" path="/var/lib/kubelet/pods/8cb744c0-bb08-4656-999b-a5957595de6c/volumes"
Sep 18 14:32:16.035055 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:16.035044    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f65a88fe-acd7-4b42-808f-81f80c051a3e" path="/var/lib/kubelet/pods/f65a88fe-acd7-4b42-808f-81f80c051a3e/volumes"
Sep 18 14:32:21.840356 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:21.840325    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:32:21.915235 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:21.915165    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:32:22.083529 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:22.083495    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn"]
Sep 18 14:32:22.083719 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:22.083698    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn" podUID="bec5d7ac-13a5-4317-b941-c8c47dbcb25e" containerName="istio-proxy" containerID="cri-o://ea7f312e95ff4116f8087b634bea4a3bdd6cd090a9e59b3ef85dd33295f32ebf" gracePeriod=30
Sep 18 14:32:22.112623 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:22.112541    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:32:27.327889 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:27.327865    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn"
Sep 18 14:32:27.441153 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:27.441093    2579 generic.go:356] "Generic (PLEG): container finished" podID="bec5d7ac-13a5-4317-b941-c8c47dbcb25e" containerID="ea7f312e95ff4116f8087b634bea4a3bdd6cd090a9e59b3ef85dd33295f32ebf" exitCode=0
Sep 18 14:32:27.441153 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:27.441134    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn"
Sep 18 14:32:27.441317 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:27.441201    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn" event={"ID":"bec5d7ac-13a5-4317-b941-c8c47dbcb25e","Type":"ContainerDied","Data":"ea7f312e95ff4116f8087b634bea4a3bdd6cd090a9e59b3ef85dd33295f32ebf"}
Sep 18 14:32:27.441317 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:27.441248    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn" event={"ID":"bec5d7ac-13a5-4317-b941-c8c47dbcb25e","Type":"ContainerDied","Data":"22d61d011891b543818332135cb7b9d5036ae72e491cd604719b8c2eb856c56b"}
Sep 18 14:32:27.441317 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:27.441273    2579 scope.go:117] "RemoveContainer" containerID="ea7f312e95ff4116f8087b634bea4a3bdd6cd090a9e59b3ef85dd33295f32ebf"
Sep 18 14:32:27.445509 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:27.445491    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-istio-envoy\") pod \"bec5d7ac-13a5-4317-b941-c8c47dbcb25e\" (UID: \"bec5d7ac-13a5-4317-b941-c8c47dbcb25e\") "
Sep 18 14:32:27.445611 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:27.445517    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-istio-data\") pod \"bec5d7ac-13a5-4317-b941-c8c47dbcb25e\" (UID: \"bec5d7ac-13a5-4317-b941-c8c47dbcb25e\") "
Sep 18 14:32:27.445611 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:27.445551    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-istiod-ca-cert\") pod \"bec5d7ac-13a5-4317-b941-c8c47dbcb25e\" (UID: \"bec5d7ac-13a5-4317-b941-c8c47dbcb25e\") "
Sep 18 14:32:27.445611 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:27.445571    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-istio-podinfo\") pod \"bec5d7ac-13a5-4317-b941-c8c47dbcb25e\" (UID: \"bec5d7ac-13a5-4317-b941-c8c47dbcb25e\") "
Sep 18 14:32:27.445611 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:27.445604    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwtmf\" (UniqueName: \"kubernetes.io/projected/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-kube-api-access-zwtmf\") pod \"bec5d7ac-13a5-4317-b941-c8c47dbcb25e\" (UID: \"bec5d7ac-13a5-4317-b941-c8c47dbcb25e\") "
Sep 18 14:32:27.445848 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:27.445630    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-istio-token\") pod \"bec5d7ac-13a5-4317-b941-c8c47dbcb25e\" (UID: \"bec5d7ac-13a5-4317-b941-c8c47dbcb25e\") "
Sep 18 14:32:27.445848 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:27.445648    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-workload-certs\") pod \"bec5d7ac-13a5-4317-b941-c8c47dbcb25e\" (UID: \"bec5d7ac-13a5-4317-b941-c8c47dbcb25e\") "
Sep 18 14:32:27.445848 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:27.445690    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-workload-socket\") pod \"bec5d7ac-13a5-4317-b941-c8c47dbcb25e\" (UID: \"bec5d7ac-13a5-4317-b941-c8c47dbcb25e\") "
Sep 18 14:32:27.445848 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:27.445727    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-credential-socket\") pod \"bec5d7ac-13a5-4317-b941-c8c47dbcb25e\" (UID: \"bec5d7ac-13a5-4317-b941-c8c47dbcb25e\") "
Sep 18 14:32:27.446058 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:27.445944    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bec5d7ac-13a5-4317-b941-c8c47dbcb25e" (UID: "bec5d7ac-13a5-4317-b941-c8c47dbcb25e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:32:27.446058 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:27.446013    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-workload-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:32:27.446228 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:27.446206    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-istio-data" (OuterVolumeSpecName: "istio-data") pod "bec5d7ac-13a5-4317-b941-c8c47dbcb25e" (UID: "bec5d7ac-13a5-4317-b941-c8c47dbcb25e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:32:27.446480 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:27.446461    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bec5d7ac-13a5-4317-b941-c8c47dbcb25e" (UID: "bec5d7ac-13a5-4317-b941-c8c47dbcb25e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:32:27.446581 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:27.446558    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bec5d7ac-13a5-4317-b941-c8c47dbcb25e" (UID: "bec5d7ac-13a5-4317-b941-c8c47dbcb25e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:32:27.446754 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:27.446719    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bec5d7ac-13a5-4317-b941-c8c47dbcb25e" (UID: "bec5d7ac-13a5-4317-b941-c8c47dbcb25e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:32:27.447876 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:27.447852    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bec5d7ac-13a5-4317-b941-c8c47dbcb25e" (UID: "bec5d7ac-13a5-4317-b941-c8c47dbcb25e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:32:27.448035 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:27.448006    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-istio-token" (OuterVolumeSpecName: "istio-token") pod "bec5d7ac-13a5-4317-b941-c8c47dbcb25e" (UID: "bec5d7ac-13a5-4317-b941-c8c47dbcb25e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:32:27.448322 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:27.448302    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-kube-api-access-zwtmf" (OuterVolumeSpecName: "kube-api-access-zwtmf") pod "bec5d7ac-13a5-4317-b941-c8c47dbcb25e" (UID: "bec5d7ac-13a5-4317-b941-c8c47dbcb25e"). InnerVolumeSpecName "kube-api-access-zwtmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:32:27.448514 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:27.448493    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bec5d7ac-13a5-4317-b941-c8c47dbcb25e" (UID: "bec5d7ac-13a5-4317-b941-c8c47dbcb25e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 14:32:27.450198 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:27.450165    2579 scope.go:117] "RemoveContainer" containerID="0141d3ef082394e73b9a341213e8f15d3f9795aa96d23ee0952356a4f0bb04bd"
Sep 18 14:32:27.462335 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:27.462320    2579 scope.go:117] "RemoveContainer" containerID="ea7f312e95ff4116f8087b634bea4a3bdd6cd090a9e59b3ef85dd33295f32ebf"
Sep 18 14:32:27.462578 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:32:27.462560    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea7f312e95ff4116f8087b634bea4a3bdd6cd090a9e59b3ef85dd33295f32ebf\": container with ID starting with ea7f312e95ff4116f8087b634bea4a3bdd6cd090a9e59b3ef85dd33295f32ebf not found: ID does not exist" containerID="ea7f312e95ff4116f8087b634bea4a3bdd6cd090a9e59b3ef85dd33295f32ebf"
Sep 18 14:32:27.462636 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:27.462585    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea7f312e95ff4116f8087b634bea4a3bdd6cd090a9e59b3ef85dd33295f32ebf"} err="failed to get container status \"ea7f312e95ff4116f8087b634bea4a3bdd6cd090a9e59b3ef85dd33295f32ebf\": rpc error: code = NotFound desc = could not find container \"ea7f312e95ff4116f8087b634bea4a3bdd6cd090a9e59b3ef85dd33295f32ebf\": container with ID starting with ea7f312e95ff4116f8087b634bea4a3bdd6cd090a9e59b3ef85dd33295f32ebf not found: ID does not exist"
Sep 18 14:32:27.462636 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:27.462602    2579 scope.go:117] "RemoveContainer" containerID="0141d3ef082394e73b9a341213e8f15d3f9795aa96d23ee0952356a4f0bb04bd"
Sep 18 14:32:27.462858 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:32:27.462841    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0141d3ef082394e73b9a341213e8f15d3f9795aa96d23ee0952356a4f0bb04bd\": container with ID starting with 0141d3ef082394e73b9a341213e8f15d3f9795aa96d23ee0952356a4f0bb04bd not found: ID does not exist" containerID="0141d3ef082394e73b9a341213e8f15d3f9795aa96d23ee0952356a4f0bb04bd"
Sep 18 14:32:27.462905 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:27.462865    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0141d3ef082394e73b9a341213e8f15d3f9795aa96d23ee0952356a4f0bb04bd"} err="failed to get container status \"0141d3ef082394e73b9a341213e8f15d3f9795aa96d23ee0952356a4f0bb04bd\": rpc error: code = NotFound desc = could not find container \"0141d3ef082394e73b9a341213e8f15d3f9795aa96d23ee0952356a4f0bb04bd\": container with ID starting with 0141d3ef082394e73b9a341213e8f15d3f9795aa96d23ee0952356a4f0bb04bd not found: ID does not exist"
Sep 18 14:32:27.546424 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:27.546400    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-credential-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:32:27.546424 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:27.546423    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-istio-envoy\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:32:27.546547 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:27.546432    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-istio-data\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:32:27.546547 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:27.546440    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-istiod-ca-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:32:27.546547 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:27.546447    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-istio-podinfo\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:32:27.546547 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:27.546455    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwtmf\" (UniqueName: \"kubernetes.io/projected/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-kube-api-access-zwtmf\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:32:27.546547 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:27.546463    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-istio-token\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:32:27.546547 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:27.546471    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec5d7ac-13a5-4317-b941-c8c47dbcb25e-workload-socket\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:32:27.762754 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:27.762728    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn"]
Sep 18 14:32:27.769101 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:27.769079    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-980661-openshift-default-59bff6f966-l5fhn"]
Sep 18 14:32:28.035991 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:28.035928    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bec5d7ac-13a5-4317-b941-c8c47dbcb25e" path="/var/lib/kubelet/pods/bec5d7ac-13a5-4317-b941-c8c47dbcb25e/volumes"
Sep 18 14:32:58.198076 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:58.198025    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:32:58.298882 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:32:58.298852    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:33:32.679745 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.679711    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-765845fd55-672tp"]
Sep 18 14:33:32.680377 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.680001    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18378eb4-074c-43fa-ab1b-9c623aba59cc" containerName="istio-proxy"
Sep 18 14:33:32.680377 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.680011    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="18378eb4-074c-43fa-ab1b-9c623aba59cc" containerName="istio-proxy"
Sep 18 14:33:32.680377 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.680024    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b95b7806-9984-4565-a427-06f996c82677" containerName="istio-proxy"
Sep 18 14:33:32.680377 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.680030    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b95b7806-9984-4565-a427-06f996c82677" containerName="istio-proxy"
Sep 18 14:33:32.680377 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.680038    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b95b7806-9984-4565-a427-06f996c82677" containerName="istio-proxy"
Sep 18 14:33:32.680377 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.680044    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b95b7806-9984-4565-a427-06f996c82677" containerName="istio-proxy"
Sep 18 14:33:32.680377 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.680049    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bec5d7ac-13a5-4317-b941-c8c47dbcb25e" containerName="istio-proxy"
Sep 18 14:33:32.680377 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.680055    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bec5d7ac-13a5-4317-b941-c8c47dbcb25e" containerName="istio-proxy"
Sep 18 14:33:32.680377 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.680061    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f65a88fe-acd7-4b42-808f-81f80c051a3e" containerName="payload-pre-processing"
Sep 18 14:33:32.680377 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.680066    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f65a88fe-acd7-4b42-808f-81f80c051a3e" containerName="payload-pre-processing"
Sep 18 14:33:32.680377 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.680072    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18378eb4-074c-43fa-ab1b-9c623aba59cc" containerName="istio-proxy"
Sep 18 14:33:32.680377 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.680077    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="18378eb4-074c-43fa-ab1b-9c623aba59cc" containerName="istio-proxy"
Sep 18 14:33:32.680377 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.680086    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bec5d7ac-13a5-4317-b941-c8c47dbcb25e" containerName="istio-proxy"
Sep 18 14:33:32.680377 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.680093    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bec5d7ac-13a5-4317-b941-c8c47dbcb25e" containerName="istio-proxy"
Sep 18 14:33:32.680377 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.680098    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0bdf88b-cc5a-4bd2-b929-6b49403b09d6" containerName="cleanup"
Sep 18 14:33:32.680377 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.680103    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0bdf88b-cc5a-4bd2-b929-6b49403b09d6" containerName="cleanup"
Sep 18 14:33:32.680377 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.680110    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e570098-8720-40dd-944b-28607af93d0a" containerName="payload-processing"
Sep 18 14:33:32.680377 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.680115    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e570098-8720-40dd-944b-28607af93d0a" containerName="payload-processing"
Sep 18 14:33:32.680377 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.680123    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46f30e53-2dbf-41f9-abd4-82701d9594fa" containerName="payload-pre-processing"
Sep 18 14:33:32.680377 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.680127    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="46f30e53-2dbf-41f9-abd4-82701d9594fa" containerName="payload-pre-processing"
Sep 18 14:33:32.680377 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.680135    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b11577e-8cc1-47f2-b262-6c2043dcafb0" containerName="payload-pre-processing"
Sep 18 14:33:32.680377 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.680140    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b11577e-8cc1-47f2-b262-6c2043dcafb0" containerName="payload-pre-processing"
Sep 18 14:33:32.680377 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.680146    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cb744c0-bb08-4656-999b-a5957595de6c" containerName="cleanup"
Sep 18 14:33:32.680377 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.680150    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cb744c0-bb08-4656-999b-a5957595de6c" containerName="cleanup"
Sep 18 14:33:32.680377 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.680210    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bec5d7ac-13a5-4317-b941-c8c47dbcb25e" containerName="istio-proxy"
Sep 18 14:33:32.680377 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.680218    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0bdf88b-cc5a-4bd2-b929-6b49403b09d6" containerName="cleanup"
Sep 18 14:33:32.680377 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.680227    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b95b7806-9984-4565-a427-06f996c82677" containerName="istio-proxy"
Sep 18 14:33:32.680377 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.680233    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cb744c0-bb08-4656-999b-a5957595de6c" containerName="cleanup"
Sep 18 14:33:32.680377 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.680238    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="46f30e53-2dbf-41f9-abd4-82701d9594fa" containerName="payload-pre-processing"
Sep 18 14:33:32.680377 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.680243    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e570098-8720-40dd-944b-28607af93d0a" containerName="payload-processing"
Sep 18 14:33:32.680377 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.680248    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f65a88fe-acd7-4b42-808f-81f80c051a3e" containerName="payload-pre-processing"
Sep 18 14:33:32.680377 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.680255    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b95b7806-9984-4565-a427-06f996c82677" containerName="istio-proxy"
Sep 18 14:33:32.680377 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.680261    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bec5d7ac-13a5-4317-b941-c8c47dbcb25e" containerName="istio-proxy"
Sep 18 14:33:32.680377 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.680267    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b11577e-8cc1-47f2-b262-6c2043dcafb0" containerName="payload-pre-processing"
Sep 18 14:33:32.680377 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.680272    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="18378eb4-074c-43fa-ab1b-9c623aba59cc" containerName="istio-proxy"
Sep 18 14:33:32.680377 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.680277    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="18378eb4-074c-43fa-ab1b-9c623aba59cc" containerName="istio-proxy"
Sep 18 14:33:32.683117 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.683097    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-765845fd55-672tp"
Sep 18 14:33:32.686404 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.686380    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-svcgm\""
Sep 18 14:33:32.686528 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.686416    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 18 14:33:32.686528 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.686417    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 18 14:33:32.686676 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.686383    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 18 14:33:32.694569 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.694546    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-765845fd55-672tp"]
Sep 18 14:33:32.717317 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.717292    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txsgj\" (UniqueName: \"kubernetes.io/projected/45bb3358-d8a4-4fa6-85b9-74f879ef2da8-kube-api-access-txsgj\") pod \"maas-controller-765845fd55-672tp\" (UID: \"45bb3358-d8a4-4fa6-85b9-74f879ef2da8\") " pod="opendatahub/maas-controller-765845fd55-672tp"
Sep 18 14:33:32.717414 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.717329    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/45bb3358-d8a4-4fa6-85b9-74f879ef2da8-webhook-cert\") pod \"maas-controller-765845fd55-672tp\" (UID: \"45bb3358-d8a4-4fa6-85b9-74f879ef2da8\") " pod="opendatahub/maas-controller-765845fd55-672tp"
Sep 18 14:33:32.717414 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.717351    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45bb3358-d8a4-4fa6-85b9-74f879ef2da8-metrics-certs\") pod \"maas-controller-765845fd55-672tp\" (UID: \"45bb3358-d8a4-4fa6-85b9-74f879ef2da8\") " pod="opendatahub/maas-controller-765845fd55-672tp"
Sep 18 14:33:32.818259 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.818238    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txsgj\" (UniqueName: \"kubernetes.io/projected/45bb3358-d8a4-4fa6-85b9-74f879ef2da8-kube-api-access-txsgj\") pod \"maas-controller-765845fd55-672tp\" (UID: \"45bb3358-d8a4-4fa6-85b9-74f879ef2da8\") " pod="opendatahub/maas-controller-765845fd55-672tp"
Sep 18 14:33:32.818360 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.818274    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/45bb3358-d8a4-4fa6-85b9-74f879ef2da8-webhook-cert\") pod \"maas-controller-765845fd55-672tp\" (UID: \"45bb3358-d8a4-4fa6-85b9-74f879ef2da8\") " pod="opendatahub/maas-controller-765845fd55-672tp"
Sep 18 14:33:32.818360 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.818294    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45bb3358-d8a4-4fa6-85b9-74f879ef2da8-metrics-certs\") pod \"maas-controller-765845fd55-672tp\" (UID: \"45bb3358-d8a4-4fa6-85b9-74f879ef2da8\") " pod="opendatahub/maas-controller-765845fd55-672tp"
Sep 18 14:33:32.820697 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.820677    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/45bb3358-d8a4-4fa6-85b9-74f879ef2da8-webhook-cert\") pod \"maas-controller-765845fd55-672tp\" (UID: \"45bb3358-d8a4-4fa6-85b9-74f879ef2da8\") " pod="opendatahub/maas-controller-765845fd55-672tp"
Sep 18 14:33:32.820837 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.820816    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45bb3358-d8a4-4fa6-85b9-74f879ef2da8-metrics-certs\") pod \"maas-controller-765845fd55-672tp\" (UID: \"45bb3358-d8a4-4fa6-85b9-74f879ef2da8\") " pod="opendatahub/maas-controller-765845fd55-672tp"
Sep 18 14:33:32.827207 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.827181    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txsgj\" (UniqueName: \"kubernetes.io/projected/45bb3358-d8a4-4fa6-85b9-74f879ef2da8-kube-api-access-txsgj\") pod \"maas-controller-765845fd55-672tp\" (UID: \"45bb3358-d8a4-4fa6-85b9-74f879ef2da8\") " pod="opendatahub/maas-controller-765845fd55-672tp"
Sep 18 14:33:32.993147 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:32.993091    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-765845fd55-672tp"
Sep 18 14:33:33.265305 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:33.265236    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-765845fd55-672tp"]
Sep 18 14:33:33.268248 ip-10-0-143-120 kubenswrapper[2579]: W0918 14:33:33.268214    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45bb3358_d8a4_4fa6_85b9_74f879ef2da8.slice/crio-2dd330e864be3dcd75f9d787f5eedb877ac300989c0bbd7ce270f54b9db78117 WatchSource:0}: Error finding container 2dd330e864be3dcd75f9d787f5eedb877ac300989c0bbd7ce270f54b9db78117: Status 404 returned error can't find the container with id 2dd330e864be3dcd75f9d787f5eedb877ac300989c0bbd7ce270f54b9db78117
Sep 18 14:33:33.746354 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:33.746318    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-765845fd55-672tp" event={"ID":"45bb3358-d8a4-4fa6-85b9-74f879ef2da8","Type":"ContainerStarted","Data":"2dd330e864be3dcd75f9d787f5eedb877ac300989c0bbd7ce270f54b9db78117"}
Sep 18 14:33:34.752070 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:34.752037    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-765845fd55-672tp" event={"ID":"45bb3358-d8a4-4fa6-85b9-74f879ef2da8","Type":"ContainerStarted","Data":"d442809c191f1d84ce993ea4dd2a8635819d3e07d46bdda6bb164f74927e2183"}
Sep 18 14:33:34.752463 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:34.752189    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-765845fd55-672tp"
Sep 18 14:33:34.771150 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:34.771106    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-765845fd55-672tp" podStartSLOduration=2.26346026 podStartE2EDuration="2.771094618s" totalImagesPullingTime="507.634358ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:33:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:33:34.769162595 +0000 UTC m=+2086.657878526" watchObservedRunningTime="2026-09-18 14:33:34.771094618 +0000 UTC m=+2086.659810549"
Sep 18 14:33:45.758286 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:45.758252    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-765845fd55-672tp"
Sep 18 14:33:50.109959 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:50.109933    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sg4tv_4626a667-4cd6-468c-8ef3-6e3a925c2608/ovn-acl-logging/0.log"
Sep 18 14:33:50.115853 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:50.115819    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sg4tv_4626a667-4cd6-468c-8ef3-6e3a925c2608/ovn-acl-logging/0.log"
Sep 18 14:33:54.988894 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:54.988861    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:33:55.074774 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:55.074746    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-765845fd55-672tp"]
Sep 18 14:33:55.074947 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:55.074926    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-765845fd55-672tp" podUID="45bb3358-d8a4-4fa6-85b9-74f879ef2da8" containerName="manager" containerID="cri-o://d442809c191f1d84ce993ea4dd2a8635819d3e07d46bdda6bb164f74927e2183" gracePeriod=30
Sep 18 14:33:55.313465 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:55.313440    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-765845fd55-672tp"
Sep 18 14:33:55.395742 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:55.395695    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txsgj\" (UniqueName: \"kubernetes.io/projected/45bb3358-d8a4-4fa6-85b9-74f879ef2da8-kube-api-access-txsgj\") pod \"45bb3358-d8a4-4fa6-85b9-74f879ef2da8\" (UID: \"45bb3358-d8a4-4fa6-85b9-74f879ef2da8\") "
Sep 18 14:33:55.395955 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:55.395773    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/45bb3358-d8a4-4fa6-85b9-74f879ef2da8-webhook-cert\") pod \"45bb3358-d8a4-4fa6-85b9-74f879ef2da8\" (UID: \"45bb3358-d8a4-4fa6-85b9-74f879ef2da8\") "
Sep 18 14:33:55.395955 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:55.395811    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45bb3358-d8a4-4fa6-85b9-74f879ef2da8-metrics-certs\") pod \"45bb3358-d8a4-4fa6-85b9-74f879ef2da8\" (UID: \"45bb3358-d8a4-4fa6-85b9-74f879ef2da8\") "
Sep 18 14:33:55.397809 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:55.397773    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45bb3358-d8a4-4fa6-85b9-74f879ef2da8-kube-api-access-txsgj" (OuterVolumeSpecName: "kube-api-access-txsgj") pod "45bb3358-d8a4-4fa6-85b9-74f879ef2da8" (UID: "45bb3358-d8a4-4fa6-85b9-74f879ef2da8"). InnerVolumeSpecName "kube-api-access-txsgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:33:55.397967 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:55.397878    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45bb3358-d8a4-4fa6-85b9-74f879ef2da8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "45bb3358-d8a4-4fa6-85b9-74f879ef2da8" (UID: "45bb3358-d8a4-4fa6-85b9-74f879ef2da8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:33:55.397967 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:55.397896    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45bb3358-d8a4-4fa6-85b9-74f879ef2da8-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "45bb3358-d8a4-4fa6-85b9-74f879ef2da8" (UID: "45bb3358-d8a4-4fa6-85b9-74f879ef2da8"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:33:55.497062 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:55.497022    2579 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/45bb3358-d8a4-4fa6-85b9-74f879ef2da8-webhook-cert\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:33:55.497062 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:55.497054    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45bb3358-d8a4-4fa6-85b9-74f879ef2da8-metrics-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:33:55.497062 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:55.497063    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txsgj\" (UniqueName: \"kubernetes.io/projected/45bb3358-d8a4-4fa6-85b9-74f879ef2da8-kube-api-access-txsgj\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:33:55.846370 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:55.846331    2579 generic.go:356] "Generic (PLEG): container finished" podID="45bb3358-d8a4-4fa6-85b9-74f879ef2da8" containerID="d442809c191f1d84ce993ea4dd2a8635819d3e07d46bdda6bb164f74927e2183" exitCode=0
Sep 18 14:33:55.846565 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:55.846381    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-765845fd55-672tp"
Sep 18 14:33:55.846565 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:55.846415    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-765845fd55-672tp" event={"ID":"45bb3358-d8a4-4fa6-85b9-74f879ef2da8","Type":"ContainerDied","Data":"d442809c191f1d84ce993ea4dd2a8635819d3e07d46bdda6bb164f74927e2183"}
Sep 18 14:33:55.846565 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:55.846471    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-765845fd55-672tp" event={"ID":"45bb3358-d8a4-4fa6-85b9-74f879ef2da8","Type":"ContainerDied","Data":"2dd330e864be3dcd75f9d787f5eedb877ac300989c0bbd7ce270f54b9db78117"}
Sep 18 14:33:55.846565 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:55.846492    2579 scope.go:117] "RemoveContainer" containerID="d442809c191f1d84ce993ea4dd2a8635819d3e07d46bdda6bb164f74927e2183"
Sep 18 14:33:55.855662 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:55.855645    2579 scope.go:117] "RemoveContainer" containerID="d442809c191f1d84ce993ea4dd2a8635819d3e07d46bdda6bb164f74927e2183"
Sep 18 14:33:55.855931 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:33:55.855911    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d442809c191f1d84ce993ea4dd2a8635819d3e07d46bdda6bb164f74927e2183\": container with ID starting with d442809c191f1d84ce993ea4dd2a8635819d3e07d46bdda6bb164f74927e2183 not found: ID does not exist" containerID="d442809c191f1d84ce993ea4dd2a8635819d3e07d46bdda6bb164f74927e2183"
Sep 18 14:33:55.855998 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:55.855941    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d442809c191f1d84ce993ea4dd2a8635819d3e07d46bdda6bb164f74927e2183"} err="failed to get container status \"d442809c191f1d84ce993ea4dd2a8635819d3e07d46bdda6bb164f74927e2183\": rpc error: code = NotFound desc = could not find container \"d442809c191f1d84ce993ea4dd2a8635819d3e07d46bdda6bb164f74927e2183\": container with ID starting with d442809c191f1d84ce993ea4dd2a8635819d3e07d46bdda6bb164f74927e2183 not found: ID does not exist"
Sep 18 14:33:55.867112 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:55.867091    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-765845fd55-672tp"]
Sep 18 14:33:55.868964 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:55.868941    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-765845fd55-672tp"]
Sep 18 14:33:56.035422 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:56.035390    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45bb3358-d8a4-4fa6-85b9-74f879ef2da8" path="/var/lib/kubelet/pods/45bb3358-d8a4-4fa6-85b9-74f879ef2da8/volumes"
Sep 18 14:33:56.558403 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:56.558368    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:33:56.673494 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:56.673468    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-765845fd55-b6jmw"]
Sep 18 14:33:56.673768 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:56.673756    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45bb3358-d8a4-4fa6-85b9-74f879ef2da8" containerName="manager"
Sep 18 14:33:56.673816 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:56.673769    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="45bb3358-d8a4-4fa6-85b9-74f879ef2da8" containerName="manager"
Sep 18 14:33:56.673850 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:56.673828    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="45bb3358-d8a4-4fa6-85b9-74f879ef2da8" containerName="manager"
Sep 18 14:33:56.678230 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:56.678215    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-765845fd55-b6jmw"
Sep 18 14:33:56.681263 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:56.681241    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 18 14:33:56.681378 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:56.681342    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-svcgm\""
Sep 18 14:33:56.681440 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:56.681425    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 18 14:33:56.681554 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:56.681541    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 18 14:33:56.688202 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:56.688165    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-765845fd55-b6jmw"]
Sep 18 14:33:56.806806 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:56.806781    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ef9b7d3-add6-4a4c-b3f3-9da526a53801-metrics-certs\") pod \"maas-controller-765845fd55-b6jmw\" (UID: \"2ef9b7d3-add6-4a4c-b3f3-9da526a53801\") " pod="opendatahub/maas-controller-765845fd55-b6jmw"
Sep 18 14:33:56.806919 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:56.806826    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2ef9b7d3-add6-4a4c-b3f3-9da526a53801-webhook-cert\") pod \"maas-controller-765845fd55-b6jmw\" (UID: \"2ef9b7d3-add6-4a4c-b3f3-9da526a53801\") " pod="opendatahub/maas-controller-765845fd55-b6jmw"
Sep 18 14:33:56.806919 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:56.806888    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gccbq\" (UniqueName: \"kubernetes.io/projected/2ef9b7d3-add6-4a4c-b3f3-9da526a53801-kube-api-access-gccbq\") pod \"maas-controller-765845fd55-b6jmw\" (UID: \"2ef9b7d3-add6-4a4c-b3f3-9da526a53801\") " pod="opendatahub/maas-controller-765845fd55-b6jmw"
Sep 18 14:33:56.907787 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:56.907728    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2ef9b7d3-add6-4a4c-b3f3-9da526a53801-webhook-cert\") pod \"maas-controller-765845fd55-b6jmw\" (UID: \"2ef9b7d3-add6-4a4c-b3f3-9da526a53801\") " pod="opendatahub/maas-controller-765845fd55-b6jmw"
Sep 18 14:33:56.907787 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:56.907766    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gccbq\" (UniqueName: \"kubernetes.io/projected/2ef9b7d3-add6-4a4c-b3f3-9da526a53801-kube-api-access-gccbq\") pod \"maas-controller-765845fd55-b6jmw\" (UID: \"2ef9b7d3-add6-4a4c-b3f3-9da526a53801\") " pod="opendatahub/maas-controller-765845fd55-b6jmw"
Sep 18 14:33:56.907950 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:56.907794    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ef9b7d3-add6-4a4c-b3f3-9da526a53801-metrics-certs\") pod \"maas-controller-765845fd55-b6jmw\" (UID: \"2ef9b7d3-add6-4a4c-b3f3-9da526a53801\") " pod="opendatahub/maas-controller-765845fd55-b6jmw"
Sep 18 14:33:56.910186 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:56.910146    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2ef9b7d3-add6-4a4c-b3f3-9da526a53801-webhook-cert\") pod \"maas-controller-765845fd55-b6jmw\" (UID: \"2ef9b7d3-add6-4a4c-b3f3-9da526a53801\") " pod="opendatahub/maas-controller-765845fd55-b6jmw"
Sep 18 14:33:56.910277 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:56.910261    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ef9b7d3-add6-4a4c-b3f3-9da526a53801-metrics-certs\") pod \"maas-controller-765845fd55-b6jmw\" (UID: \"2ef9b7d3-add6-4a4c-b3f3-9da526a53801\") " pod="opendatahub/maas-controller-765845fd55-b6jmw"
Sep 18 14:33:56.915025 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:56.915004    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gccbq\" (UniqueName: \"kubernetes.io/projected/2ef9b7d3-add6-4a4c-b3f3-9da526a53801-kube-api-access-gccbq\") pod \"maas-controller-765845fd55-b6jmw\" (UID: \"2ef9b7d3-add6-4a4c-b3f3-9da526a53801\") " pod="opendatahub/maas-controller-765845fd55-b6jmw"
Sep 18 14:33:56.987570 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:56.987547    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-765845fd55-b6jmw"
Sep 18 14:33:57.259861 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:57.259607    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-765845fd55-b6jmw"]
Sep 18 14:33:57.262376 ip-10-0-143-120 kubenswrapper[2579]: W0918 14:33:57.262340    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ef9b7d3_add6_4a4c_b3f3_9da526a53801.slice/crio-901c513d4d42df7162b54a8cf1e34a60e2562ddde84e8df703c300534f826df4 WatchSource:0}: Error finding container 901c513d4d42df7162b54a8cf1e34a60e2562ddde84e8df703c300534f826df4: Status 404 returned error can't find the container with id 901c513d4d42df7162b54a8cf1e34a60e2562ddde84e8df703c300534f826df4
Sep 18 14:33:57.858675 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:57.858641    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-765845fd55-b6jmw" event={"ID":"2ef9b7d3-add6-4a4c-b3f3-9da526a53801","Type":"ContainerStarted","Data":"4f6195a0bf607ca363d935c28a89102fb0a3d3cea3f41f3ddd7a7c833cc715c3"}
Sep 18 14:33:57.858834 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:57.858690    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-765845fd55-b6jmw" event={"ID":"2ef9b7d3-add6-4a4c-b3f3-9da526a53801","Type":"ContainerStarted","Data":"901c513d4d42df7162b54a8cf1e34a60e2562ddde84e8df703c300534f826df4"}
Sep 18 14:33:57.858834 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:57.858720    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-765845fd55-b6jmw"
Sep 18 14:33:57.878504 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:57.878427    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-765845fd55-b6jmw" podStartSLOduration=1.5234403890000001 podStartE2EDuration="1.878412281s" totalImagesPullingTime="354.971892ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:33:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:33:57.876158309 +0000 UTC m=+2109.764874241" watchObservedRunningTime="2026-09-18 14:33:57.878412281 +0000 UTC m=+2109.767128212"
Sep 18 14:33:58.869149 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:33:58.869120    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:34:08.865294 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:34:08.865250    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-765845fd55-b6jmw"
Sep 18 14:35:15.469099 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:35:15.469060    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xcjrd"]
Sep 18 14:35:15.469652 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:35:15.469290    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xcjrd" podUID="c2973f4c-bada-49b8-a745-532a586bd54a" containerName="manager" containerID="cri-o://693d26616f6d644ab0f256340ce43871a6139401c9909c2da6c8251252e58e93" gracePeriod=10
Sep 18 14:35:17.217672 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:35:17.217641    2579 generic.go:356] "Generic (PLEG): container finished" podID="c2973f4c-bada-49b8-a745-532a586bd54a" containerID="693d26616f6d644ab0f256340ce43871a6139401c9909c2da6c8251252e58e93" exitCode=0
Sep 18 14:35:17.218042 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:35:17.217678    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xcjrd" event={"ID":"c2973f4c-bada-49b8-a745-532a586bd54a","Type":"ContainerDied","Data":"693d26616f6d644ab0f256340ce43871a6139401c9909c2da6c8251252e58e93"}
Sep 18 14:35:17.306488 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:35:17.306460    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xcjrd"
Sep 18 14:35:17.418131 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:35:17.418098    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c2973f4c-bada-49b8-a745-532a586bd54a-extensions-socket-volume\") pod \"c2973f4c-bada-49b8-a745-532a586bd54a\" (UID: \"c2973f4c-bada-49b8-a745-532a586bd54a\") "
Sep 18 14:35:17.418317 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:35:17.418166    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qlszs\" (UniqueName: \"kubernetes.io/projected/c2973f4c-bada-49b8-a745-532a586bd54a-kube-api-access-qlszs\") pod \"c2973f4c-bada-49b8-a745-532a586bd54a\" (UID: \"c2973f4c-bada-49b8-a745-532a586bd54a\") "
Sep 18 14:35:17.418450 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:35:17.418427    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2973f4c-bada-49b8-a745-532a586bd54a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c2973f4c-bada-49b8-a745-532a586bd54a" (UID: "c2973f4c-bada-49b8-a745-532a586bd54a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:35:17.420128 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:35:17.420108    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2973f4c-bada-49b8-a745-532a586bd54a-kube-api-access-qlszs" (OuterVolumeSpecName: "kube-api-access-qlszs") pod "c2973f4c-bada-49b8-a745-532a586bd54a" (UID: "c2973f4c-bada-49b8-a745-532a586bd54a"). InnerVolumeSpecName "kube-api-access-qlszs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:35:17.519676 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:35:17.519614    2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c2973f4c-bada-49b8-a745-532a586bd54a-extensions-socket-volume\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:35:17.519676 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:35:17.519639    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlszs\" (UniqueName: \"kubernetes.io/projected/c2973f4c-bada-49b8-a745-532a586bd54a-kube-api-access-qlszs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:35:18.229381 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:35:18.229354    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xcjrd"
Sep 18 14:35:18.229788 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:35:18.229374    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xcjrd" event={"ID":"c2973f4c-bada-49b8-a745-532a586bd54a","Type":"ContainerDied","Data":"ac7136b2b794a37f21e11f750f8094175bb31836822dbf4203115fd77b190cc6"}
Sep 18 14:35:18.229788 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:35:18.229449    2579 scope.go:117] "RemoveContainer" containerID="693d26616f6d644ab0f256340ce43871a6139401c9909c2da6c8251252e58e93"
Sep 18 14:35:18.246892 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:35:18.246872    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xcjrd"]
Sep 18 14:35:18.250966 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:35:18.250942    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xcjrd"]
Sep 18 14:35:20.035423 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:35:20.035378    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2973f4c-bada-49b8-a745-532a586bd54a" path="/var/lib/kubelet/pods/c2973f4c-bada-49b8-a745-532a586bd54a/volumes"
Sep 18 14:35:28.468593 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:35:28.468539    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:35:29.718650 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:35:29.718611    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:36:12.057869 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:36:12.057825    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:36:16.137688 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:36:16.137655    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bqcwt"]
Sep 18 14:36:16.138065 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:36:16.137922    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2973f4c-bada-49b8-a745-532a586bd54a" containerName="manager"
Sep 18 14:36:16.138065 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:36:16.137932    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2973f4c-bada-49b8-a745-532a586bd54a" containerName="manager"
Sep 18 14:36:16.138065 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:36:16.137981    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2973f4c-bada-49b8-a745-532a586bd54a" containerName="manager"
Sep 18 14:36:16.140709 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:36:16.140693    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bqcwt"
Sep 18 14:36:16.143486 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:36:16.143461    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-q692n\""
Sep 18 14:36:16.155011 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:36:16.154988    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bqcwt"]
Sep 18 14:36:16.245917 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:36:16.245885    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmtsr\" (UniqueName: \"kubernetes.io/projected/eea2ba96-deb9-4352-b79e-d95cd15ec9ae-kube-api-access-gmtsr\") pod \"kuadrant-operator-controller-manager-5769f99d9f-bqcwt\" (UID: \"eea2ba96-deb9-4352-b79e-d95cd15ec9ae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bqcwt"
Sep 18 14:36:16.246036 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:36:16.245947    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eea2ba96-deb9-4352-b79e-d95cd15ec9ae-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-bqcwt\" (UID: \"eea2ba96-deb9-4352-b79e-d95cd15ec9ae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bqcwt"
Sep 18 14:36:16.347196 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:36:16.347159    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eea2ba96-deb9-4352-b79e-d95cd15ec9ae-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-bqcwt\" (UID: \"eea2ba96-deb9-4352-b79e-d95cd15ec9ae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bqcwt"
Sep 18 14:36:16.347308 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:36:16.347207    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmtsr\" (UniqueName: \"kubernetes.io/projected/eea2ba96-deb9-4352-b79e-d95cd15ec9ae-kube-api-access-gmtsr\") pod \"kuadrant-operator-controller-manager-5769f99d9f-bqcwt\" (UID: \"eea2ba96-deb9-4352-b79e-d95cd15ec9ae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bqcwt"
Sep 18 14:36:16.347591 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:36:16.347572    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eea2ba96-deb9-4352-b79e-d95cd15ec9ae-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-bqcwt\" (UID: \"eea2ba96-deb9-4352-b79e-d95cd15ec9ae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bqcwt"
Sep 18 14:36:16.355114 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:36:16.355090    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmtsr\" (UniqueName: \"kubernetes.io/projected/eea2ba96-deb9-4352-b79e-d95cd15ec9ae-kube-api-access-gmtsr\") pod \"kuadrant-operator-controller-manager-5769f99d9f-bqcwt\" (UID: \"eea2ba96-deb9-4352-b79e-d95cd15ec9ae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bqcwt"
Sep 18 14:36:16.450331 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:36:16.450277    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bqcwt"
Sep 18 14:36:16.725095 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:36:16.725071    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bqcwt"]
Sep 18 14:36:16.727735 ip-10-0-143-120 kubenswrapper[2579]: W0918 14:36:16.727706    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeea2ba96_deb9_4352_b79e_d95cd15ec9ae.slice/crio-7efcdf644032ba38da39990c04b2fb6e97a72f782f75cc5991b37a680ea6255c WatchSource:0}: Error finding container 7efcdf644032ba38da39990c04b2fb6e97a72f782f75cc5991b37a680ea6255c: Status 404 returned error can't find the container with id 7efcdf644032ba38da39990c04b2fb6e97a72f782f75cc5991b37a680ea6255c
Sep 18 14:36:17.487974 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:36:17.487934    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bqcwt" event={"ID":"eea2ba96-deb9-4352-b79e-d95cd15ec9ae","Type":"ContainerStarted","Data":"24b1c162027a1d11d775151ced3a42c241a7bc95dea1f61f69e211af19268ed5"}
Sep 18 14:36:17.488387 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:36:17.487994    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bqcwt" event={"ID":"eea2ba96-deb9-4352-b79e-d95cd15ec9ae","Type":"ContainerStarted","Data":"7efcdf644032ba38da39990c04b2fb6e97a72f782f75cc5991b37a680ea6255c"}
Sep 18 14:36:17.488387 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:36:17.488120    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bqcwt"
Sep 18 14:36:17.513983 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:36:17.513919    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bqcwt" podStartSLOduration=1.51390335 podStartE2EDuration="1.51390335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:36:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:36:17.512398499 +0000 UTC m=+2249.401114424" watchObservedRunningTime="2026-09-18 14:36:17.51390335 +0000 UTC m=+2249.402619281"
Sep 18 14:36:28.495251 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:36:28.495217    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bqcwt"
Sep 18 14:36:47.554628 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:36:47.554581    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:36:50.313525 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:36:50.313491    2579 scope.go:117] "RemoveContainer" containerID="4de0e1a6410693625c206767d157594ea1648a7187daa755bd4368412679b53b"
Sep 18 14:36:50.321131 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:36:50.321113    2579 scope.go:117] "RemoveContainer" containerID="668048be0ff704928eff6542a96751ebc5ae70fe0a486a2d39de9b8a51ccdd03"
Sep 18 14:36:56.172444 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:36:56.172403    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:37:15.122833 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:37:15.122793    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:37:30.335839 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:37:30.335808    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:37:48.216969 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:37:48.216845    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:37:48.980520 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:37:48.980477    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:37:56.855739 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:37:56.855697    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:38:08.202912 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:08.202873    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-697bd64575-z797g"]
Sep 18 14:38:08.206505 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:08.206488    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-697bd64575-z797g"
Sep 18 14:38:08.209324 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:38:08.209292    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"maas-api-697bd64575-z797g\" is forbidden: User \"system:node:ip-10-0-143-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-143-120.ec2.internal' and this object" podUID="f94ef533-929c-4547-be99-bded167d6857" pod="odh-ai-gateway-infra/maas-api-697bd64575-z797g"
Sep 18 14:38:08.238473 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:08.238438    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-697bd64575-z797g"]
Sep 18 14:38:08.391499 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:08.391468    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f94ef533-929c-4547-be99-bded167d6857-maas-api-tls\") pod \"maas-api-697bd64575-z797g\" (UID: \"f94ef533-929c-4547-be99-bded167d6857\") " pod="odh-ai-gateway-infra/maas-api-697bd64575-z797g"
Sep 18 14:38:08.391499 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:08.391500    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p82wt\" (UniqueName: \"kubernetes.io/projected/f94ef533-929c-4547-be99-bded167d6857-kube-api-access-p82wt\") pod \"maas-api-697bd64575-z797g\" (UID: \"f94ef533-929c-4547-be99-bded167d6857\") " pod="odh-ai-gateway-infra/maas-api-697bd64575-z797g"
Sep 18 14:38:08.391721 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:08.391556    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f94ef533-929c-4547-be99-bded167d6857-openshift-service-ca\") pod \"maas-api-697bd64575-z797g\" (UID: \"f94ef533-929c-4547-be99-bded167d6857\") " pod="odh-ai-gateway-infra/maas-api-697bd64575-z797g"
Sep 18 14:38:08.391721 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:08.391579    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f94ef533-929c-4547-be99-bded167d6857-metrics-certs\") pod \"maas-api-697bd64575-z797g\" (UID: \"f94ef533-929c-4547-be99-bded167d6857\") " pod="odh-ai-gateway-infra/maas-api-697bd64575-z797g"
Sep 18 14:38:08.492433 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:08.492353    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f94ef533-929c-4547-be99-bded167d6857-openshift-service-ca\") pod \"maas-api-697bd64575-z797g\" (UID: \"f94ef533-929c-4547-be99-bded167d6857\") " pod="odh-ai-gateway-infra/maas-api-697bd64575-z797g"
Sep 18 14:38:08.492433 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:08.492395    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f94ef533-929c-4547-be99-bded167d6857-metrics-certs\") pod \"maas-api-697bd64575-z797g\" (UID: \"f94ef533-929c-4547-be99-bded167d6857\") " pod="odh-ai-gateway-infra/maas-api-697bd64575-z797g"
Sep 18 14:38:08.492433 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:08.492430    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f94ef533-929c-4547-be99-bded167d6857-maas-api-tls\") pod \"maas-api-697bd64575-z797g\" (UID: \"f94ef533-929c-4547-be99-bded167d6857\") " pod="odh-ai-gateway-infra/maas-api-697bd64575-z797g"
Sep 18 14:38:08.492723 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:08.492457    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p82wt\" (UniqueName: \"kubernetes.io/projected/f94ef533-929c-4547-be99-bded167d6857-kube-api-access-p82wt\") pod \"maas-api-697bd64575-z797g\" (UID: \"f94ef533-929c-4547-be99-bded167d6857\") " pod="odh-ai-gateway-infra/maas-api-697bd64575-z797g"
Sep 18 14:38:08.493092 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:08.493065    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f94ef533-929c-4547-be99-bded167d6857-openshift-service-ca\") pod \"maas-api-697bd64575-z797g\" (UID: \"f94ef533-929c-4547-be99-bded167d6857\") " pod="odh-ai-gateway-infra/maas-api-697bd64575-z797g"
Sep 18 14:38:08.494930 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:08.494906    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f94ef533-929c-4547-be99-bded167d6857-maas-api-tls\") pod \"maas-api-697bd64575-z797g\" (UID: \"f94ef533-929c-4547-be99-bded167d6857\") " pod="odh-ai-gateway-infra/maas-api-697bd64575-z797g"
Sep 18 14:38:08.495015 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:08.494953    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f94ef533-929c-4547-be99-bded167d6857-metrics-certs\") pod \"maas-api-697bd64575-z797g\" (UID: \"f94ef533-929c-4547-be99-bded167d6857\") " pod="odh-ai-gateway-infra/maas-api-697bd64575-z797g"
Sep 18 14:38:08.508545 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:08.508525    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p82wt\" (UniqueName: \"kubernetes.io/projected/f94ef533-929c-4547-be99-bded167d6857-kube-api-access-p82wt\") pod \"maas-api-697bd64575-z797g\" (UID: \"f94ef533-929c-4547-be99-bded167d6857\") " pod="odh-ai-gateway-infra/maas-api-697bd64575-z797g"
Sep 18 14:38:08.516311 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:08.516286    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-697bd64575-z797g"
Sep 18 14:38:08.792229 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:08.792149    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-697bd64575-z797g"]
Sep 18 14:38:08.797727 ip-10-0-143-120 kubenswrapper[2579]: W0918 14:38:08.797697    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf94ef533_929c_4547_be99_bded167d6857.slice/crio-cabaf582411e68db0438c52bd1a04ac2d0470604b80e08cc84bc1f587ab42f8f WatchSource:0}: Error finding container cabaf582411e68db0438c52bd1a04ac2d0470604b80e08cc84bc1f587ab42f8f: Status 404 returned error can't find the container with id cabaf582411e68db0438c52bd1a04ac2d0470604b80e08cc84bc1f587ab42f8f
Sep 18 14:38:08.799012 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:08.798992    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 14:38:08.973615 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:08.973587    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-697bd64575-z797g" event={"ID":"f94ef533-929c-4547-be99-bded167d6857","Type":"ContainerStarted","Data":"cabaf582411e68db0438c52bd1a04ac2d0470604b80e08cc84bc1f587ab42f8f"}
Sep 18 14:38:09.979643 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:09.979609    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-697bd64575-z797g" event={"ID":"f94ef533-929c-4547-be99-bded167d6857","Type":"ContainerStarted","Data":"11964d1b94ca45fa814426184285c8f4f6e365188e90f8830963041b62483a1b"}
Sep 18 14:38:09.980001 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:09.979714    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-697bd64575-z797g"
Sep 18 14:38:15.989460 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:15.989432    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-697bd64575-z797g"
Sep 18 14:38:16.047112 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:16.047065    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-697bd64575-z797g" podStartSLOduration=7.469365858 podStartE2EDuration="8.047050344s" totalImagesPullingTime="577.684486ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:38:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:38:10.014854528 +0000 UTC m=+2361.903570459" watchObservedRunningTime="2026-09-18 14:38:16.047050344 +0000 UTC m=+2367.935766275"
Sep 18 14:38:16.119854 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:16.119827    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-55f6c99747-pv262"]
Sep 18 14:38:16.120048 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:16.120028    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-55f6c99747-pv262" podUID="d2c22a5f-b11c-416b-936c-ea0fd28c4614" containerName="maas-api" containerID="cri-o://7b0298d1aab5148f9001ec4e463a9d565222941605826313496fb0c4102f1754" gracePeriod=30
Sep 18 14:38:17.364432 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:17.364394    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55f6c99747-pv262"
Sep 18 14:38:17.461022 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:17.460947    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9cmv\" (UniqueName: \"kubernetes.io/projected/d2c22a5f-b11c-416b-936c-ea0fd28c4614-kube-api-access-t9cmv\") pod \"d2c22a5f-b11c-416b-936c-ea0fd28c4614\" (UID: \"d2c22a5f-b11c-416b-936c-ea0fd28c4614\") "
Sep 18 14:38:17.461202 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:17.461031    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2c22a5f-b11c-416b-936c-ea0fd28c4614-openshift-service-ca\") pod \"d2c22a5f-b11c-416b-936c-ea0fd28c4614\" (UID: \"d2c22a5f-b11c-416b-936c-ea0fd28c4614\") "
Sep 18 14:38:17.461202 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:17.461058    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2c22a5f-b11c-416b-936c-ea0fd28c4614-metrics-certs\") pod \"d2c22a5f-b11c-416b-936c-ea0fd28c4614\" (UID: \"d2c22a5f-b11c-416b-936c-ea0fd28c4614\") "
Sep 18 14:38:17.461202 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:17.461121    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2c22a5f-b11c-416b-936c-ea0fd28c4614-maas-api-tls\") pod \"d2c22a5f-b11c-416b-936c-ea0fd28c4614\" (UID: \"d2c22a5f-b11c-416b-936c-ea0fd28c4614\") "
Sep 18 14:38:17.461474 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:17.461438    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2c22a5f-b11c-416b-936c-ea0fd28c4614-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d2c22a5f-b11c-416b-936c-ea0fd28c4614" (UID: "d2c22a5f-b11c-416b-936c-ea0fd28c4614"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:38:17.463239 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:17.463197    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2c22a5f-b11c-416b-936c-ea0fd28c4614-kube-api-access-t9cmv" (OuterVolumeSpecName: "kube-api-access-t9cmv") pod "d2c22a5f-b11c-416b-936c-ea0fd28c4614" (UID: "d2c22a5f-b11c-416b-936c-ea0fd28c4614"). InnerVolumeSpecName "kube-api-access-t9cmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:38:17.463357 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:17.463276    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2c22a5f-b11c-416b-936c-ea0fd28c4614-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d2c22a5f-b11c-416b-936c-ea0fd28c4614" (UID: "d2c22a5f-b11c-416b-936c-ea0fd28c4614"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:38:17.463357 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:17.463325    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2c22a5f-b11c-416b-936c-ea0fd28c4614-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d2c22a5f-b11c-416b-936c-ea0fd28c4614" (UID: "d2c22a5f-b11c-416b-936c-ea0fd28c4614"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:38:17.561905 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:17.561877    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2c22a5f-b11c-416b-936c-ea0fd28c4614-openshift-service-ca\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:38:17.561905 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:17.561899    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2c22a5f-b11c-416b-936c-ea0fd28c4614-metrics-certs\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:38:17.561905 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:17.561908    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2c22a5f-b11c-416b-936c-ea0fd28c4614-maas-api-tls\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:38:17.562092 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:17.561916    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9cmv\" (UniqueName: \"kubernetes.io/projected/d2c22a5f-b11c-416b-936c-ea0fd28c4614-kube-api-access-t9cmv\") on node \"ip-10-0-143-120.ec2.internal\" DevicePath \"\""
Sep 18 14:38:18.015154 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:18.015124    2579 generic.go:356] "Generic (PLEG): container finished" podID="d2c22a5f-b11c-416b-936c-ea0fd28c4614" containerID="7b0298d1aab5148f9001ec4e463a9d565222941605826313496fb0c4102f1754" exitCode=0
Sep 18 14:38:18.015365 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:18.015160    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55f6c99747-pv262" event={"ID":"d2c22a5f-b11c-416b-936c-ea0fd28c4614","Type":"ContainerDied","Data":"7b0298d1aab5148f9001ec4e463a9d565222941605826313496fb0c4102f1754"}
Sep 18 14:38:18.015365 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:18.015189    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55f6c99747-pv262"
Sep 18 14:38:18.015365 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:18.015212    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55f6c99747-pv262" event={"ID":"d2c22a5f-b11c-416b-936c-ea0fd28c4614","Type":"ContainerDied","Data":"e52a4314f6522a424b7b40468c28b55084f3f4139c6e84b6cbcf6d2b98a0b9dd"}
Sep 18 14:38:18.015365 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:18.015234    2579 scope.go:117] "RemoveContainer" containerID="7b0298d1aab5148f9001ec4e463a9d565222941605826313496fb0c4102f1754"
Sep 18 14:38:18.023883 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:18.023866    2579 scope.go:117] "RemoveContainer" containerID="7b0298d1aab5148f9001ec4e463a9d565222941605826313496fb0c4102f1754"
Sep 18 14:38:18.024151 ip-10-0-143-120 kubenswrapper[2579]: E0918 14:38:18.024129    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b0298d1aab5148f9001ec4e463a9d565222941605826313496fb0c4102f1754\": container with ID starting with 7b0298d1aab5148f9001ec4e463a9d565222941605826313496fb0c4102f1754 not found: ID does not exist" containerID="7b0298d1aab5148f9001ec4e463a9d565222941605826313496fb0c4102f1754"
Sep 18 14:38:18.024259 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:18.024159    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b0298d1aab5148f9001ec4e463a9d565222941605826313496fb0c4102f1754"} err="failed to get container status \"7b0298d1aab5148f9001ec4e463a9d565222941605826313496fb0c4102f1754\": rpc error: code = NotFound desc = could not find container \"7b0298d1aab5148f9001ec4e463a9d565222941605826313496fb0c4102f1754\": container with ID starting with 7b0298d1aab5148f9001ec4e463a9d565222941605826313496fb0c4102f1754 not found: ID does not exist"
Sep 18 14:38:18.039905 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:18.039881    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-55f6c99747-pv262"]
Sep 18 14:38:18.043754 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:18.043730    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-55f6c99747-pv262"]
Sep 18 14:38:20.034150 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:20.034115    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2c22a5f-b11c-416b-936c-ea0fd28c4614" path="/var/lib/kubelet/pods/d2c22a5f-b11c-416b-936c-ea0fd28c4614/volumes"
Sep 18 14:38:38.361117 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:38.361087    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-c8788b85-pdmmn_a8cc30de-f512-4a5f-90c0-1ff70103cbcd/authorino/0.log"
Sep 18 14:38:45.194617 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:45.194583    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-l89sx_413479c5-b502-4549-8b20-852bc9d3648f/manager/0.log"
Sep 18 14:38:45.333529 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:45.333504    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-765845fd55-b6jmw_2ef9b7d3-add6-4a4c-b3f3-9da526a53801/manager/0.log"
Sep 18 14:38:45.707333 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:45.707307    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-74d96fdcf9-6h2ct_e3ab8f69-7fd2-46e0-94fb-98db7182cde6/copy-manifests/0.log"
Sep 18 14:38:45.715029 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:45.715004    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-74d96fdcf9-6h2ct_e3ab8f69-7fd2-46e0-94fb-98db7182cde6/manager/0.log"
Sep 18 14:38:46.450737 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:46.450705    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-697bd64575-z797g_f94ef533-929c-4547-be99-bded167d6857/maas-api/0.log"
Sep 18 14:38:47.555459 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:47.555437    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-c8788b85-pdmmn_a8cc30de-f512-4a5f-90c0-1ff70103cbcd/authorino/0.log"
Sep 18 14:38:47.673405 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:47.673380    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-c8cdh_769de1a3-059d-4206-92e2-83379db6da05/manager/0.log"
Sep 18 14:38:47.795245 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:47.795223    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-m7gl9_9e657c79-304e-412c-88b5-9f694e8fae47/manager/0.log"
Sep 18 14:38:48.040244 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:48.040181    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-bqcwt_eea2ba96-deb9-4352-b79e-d95cd15ec9ae/manager/0.log"
Sep 18 14:38:48.794154 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:48.794120    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-vklds_e4574081-5495-4cd9-8c39-0c7cded44228/discovery/0.log"
Sep 18 14:38:49.045542 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:49.045460    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86dc75d4b5-ls4xg_6846cc5d-9e5f-4a5a-abbb-54c7163c1f96/kube-auth-proxy/0.log"
Sep 18 14:38:49.923900 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:49.923871    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-6gb2n_05d9cb17-23c2-41d1-aa47-3d7b7588a3bd/main/0.log"
Sep 18 14:38:50.132860 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:50.132837    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sg4tv_4626a667-4cd6-468c-8ef3-6e3a925c2608/ovn-acl-logging/0.log"
Sep 18 14:38:50.140368 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:50.140349    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sg4tv_4626a667-4cd6-468c-8ef3-6e3a925c2608/ovn-acl-logging/0.log"
Sep 18 14:38:50.178526 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:50.178436    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-bj7wf_901a17ec-cc37-47c1-bb69-29c51f62e87e/main/0.log"
Sep 18 14:38:56.971472 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:56.971436    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:38:57.106667 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:38:57.106627    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log" limit=1024
Sep 18 14:39:00.196823 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:00.196765    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2hvmm_ca72836e-7ed0-47e5-a106-2f47340eb6b1/global-pull-secret-syncer/0.log"
Sep 18 14:39:00.398547 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:00.398516    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-fkrgg_9f51a32f-f091-49b2-97f8-7e3ba0008474/konnectivity-agent/0.log"
Sep 18 14:39:00.653606 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:00.653574    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-120.ec2.internal_59194d24519fb418bb58aa767b6c58b0/haproxy/0.log"
Sep 18 14:39:04.848294 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:04.848224    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-c8788b85-pdmmn_a8cc30de-f512-4a5f-90c0-1ff70103cbcd/authorino/0.log"
Sep 18 14:39:04.897398 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:04.897372    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-c8cdh_769de1a3-059d-4206-92e2-83379db6da05/manager/0.log"
Sep 18 14:39:04.930696 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:04.930675    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-m7gl9_9e657c79-304e-412c-88b5-9f694e8fae47/manager/0.log"
Sep 18 14:39:04.998845 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:04.998824    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-bqcwt_eea2ba96-deb9-4352-b79e-d95cd15ec9ae/manager/0.log"
Sep 18 14:39:07.814210 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:07.814167    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x4bcs_d3a439e0-fb98-4dab-a637-63d2d17ae8b1/node-exporter/0.log"
Sep 18 14:39:07.878715 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:07.878689    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x4bcs_d3a439e0-fb98-4dab-a637-63d2d17ae8b1/kube-rbac-proxy/0.log"
Sep 18 14:39:07.945306 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:07.945280    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x4bcs_d3a439e0-fb98-4dab-a637-63d2d17ae8b1/init-textfile/0.log"
Sep 18 14:39:09.859490 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:09.859445    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9xtsh/perf-node-gather-daemonset-zmdm6"]
Sep 18 14:39:09.860060 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:09.859856    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2c22a5f-b11c-416b-936c-ea0fd28c4614" containerName="maas-api"
Sep 18 14:39:09.860060 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:09.859875    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2c22a5f-b11c-416b-936c-ea0fd28c4614" containerName="maas-api"
Sep 18 14:39:09.860060 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:09.859939    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2c22a5f-b11c-416b-936c-ea0fd28c4614" containerName="maas-api"
Sep 18 14:39:09.864261 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:09.864238    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9xtsh/perf-node-gather-daemonset-zmdm6"
Sep 18 14:39:09.866873 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:09.866844    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9xtsh\"/\"default-dockercfg-r9lr6\""
Sep 18 14:39:09.867007 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:09.866903    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9xtsh\"/\"openshift-service-ca.crt\""
Sep 18 14:39:09.867643 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:09.867619    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9xtsh\"/\"kube-root-ca.crt\""
Sep 18 14:39:09.885215 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:09.885183    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9xtsh/perf-node-gather-daemonset-zmdm6"]
Sep 18 14:39:09.951738 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:09.951712    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/55cffdc7-1e6c-4cfe-a016-2eb0df1a37d3-proc\") pod \"perf-node-gather-daemonset-zmdm6\" (UID: \"55cffdc7-1e6c-4cfe-a016-2eb0df1a37d3\") " pod="openshift-must-gather-9xtsh/perf-node-gather-daemonset-zmdm6"
Sep 18 14:39:09.951879 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:09.951745    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/55cffdc7-1e6c-4cfe-a016-2eb0df1a37d3-sys\") pod \"perf-node-gather-daemonset-zmdm6\" (UID: \"55cffdc7-1e6c-4cfe-a016-2eb0df1a37d3\") " pod="openshift-must-gather-9xtsh/perf-node-gather-daemonset-zmdm6"
Sep 18 14:39:09.951879 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:09.951832    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgdbt\" (UniqueName: \"kubernetes.io/projected/55cffdc7-1e6c-4cfe-a016-2eb0df1a37d3-kube-api-access-fgdbt\") pod \"perf-node-gather-daemonset-zmdm6\" (UID: \"55cffdc7-1e6c-4cfe-a016-2eb0df1a37d3\") " pod="openshift-must-gather-9xtsh/perf-node-gather-daemonset-zmdm6"
Sep 18 14:39:09.951973 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:09.951895    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/55cffdc7-1e6c-4cfe-a016-2eb0df1a37d3-podres\") pod \"perf-node-gather-daemonset-zmdm6\" (UID: \"55cffdc7-1e6c-4cfe-a016-2eb0df1a37d3\") " pod="openshift-must-gather-9xtsh/perf-node-gather-daemonset-zmdm6"
Sep 18 14:39:09.951973 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:09.951920    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/55cffdc7-1e6c-4cfe-a016-2eb0df1a37d3-lib-modules\") pod \"perf-node-gather-daemonset-zmdm6\" (UID: \"55cffdc7-1e6c-4cfe-a016-2eb0df1a37d3\") " pod="openshift-must-gather-9xtsh/perf-node-gather-daemonset-zmdm6"
Sep 18 14:39:10.053015 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:10.052988    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/55cffdc7-1e6c-4cfe-a016-2eb0df1a37d3-proc\") pod \"perf-node-gather-daemonset-zmdm6\" (UID: \"55cffdc7-1e6c-4cfe-a016-2eb0df1a37d3\") " pod="openshift-must-gather-9xtsh/perf-node-gather-daemonset-zmdm6"
Sep 18 14:39:10.053147 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:10.053019    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/55cffdc7-1e6c-4cfe-a016-2eb0df1a37d3-sys\") pod \"perf-node-gather-daemonset-zmdm6\" (UID: \"55cffdc7-1e6c-4cfe-a016-2eb0df1a37d3\") " pod="openshift-must-gather-9xtsh/perf-node-gather-daemonset-zmdm6"
Sep 18 14:39:10.053147 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:10.053056    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgdbt\" (UniqueName: \"kubernetes.io/projected/55cffdc7-1e6c-4cfe-a016-2eb0df1a37d3-kube-api-access-fgdbt\") pod \"perf-node-gather-daemonset-zmdm6\" (UID: \"55cffdc7-1e6c-4cfe-a016-2eb0df1a37d3\") " pod="openshift-must-gather-9xtsh/perf-node-gather-daemonset-zmdm6"
Sep 18 14:39:10.053147 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:10.053105    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/55cffdc7-1e6c-4cfe-a016-2eb0df1a37d3-proc\") pod \"perf-node-gather-daemonset-zmdm6\" (UID: \"55cffdc7-1e6c-4cfe-a016-2eb0df1a37d3\") " pod="openshift-must-gather-9xtsh/perf-node-gather-daemonset-zmdm6"
Sep 18 14:39:10.053147 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:10.053128    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/55cffdc7-1e6c-4cfe-a016-2eb0df1a37d3-sys\") pod \"perf-node-gather-daemonset-zmdm6\" (UID: \"55cffdc7-1e6c-4cfe-a016-2eb0df1a37d3\") " pod="openshift-must-gather-9xtsh/perf-node-gather-daemonset-zmdm6"
Sep 18 14:39:10.053418 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:10.053200    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/55cffdc7-1e6c-4cfe-a016-2eb0df1a37d3-podres\") pod \"perf-node-gather-daemonset-zmdm6\" (UID: \"55cffdc7-1e6c-4cfe-a016-2eb0df1a37d3\") " pod="openshift-must-gather-9xtsh/perf-node-gather-daemonset-zmdm6"
Sep 18 14:39:10.053418 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:10.053230    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/55cffdc7-1e6c-4cfe-a016-2eb0df1a37d3-lib-modules\") pod \"perf-node-gather-daemonset-zmdm6\" (UID: \"55cffdc7-1e6c-4cfe-a016-2eb0df1a37d3\") " pod="openshift-must-gather-9xtsh/perf-node-gather-daemonset-zmdm6"
Sep 18 14:39:10.053418 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:10.053314    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/55cffdc7-1e6c-4cfe-a016-2eb0df1a37d3-podres\") pod \"perf-node-gather-daemonset-zmdm6\" (UID: \"55cffdc7-1e6c-4cfe-a016-2eb0df1a37d3\") " pod="openshift-must-gather-9xtsh/perf-node-gather-daemonset-zmdm6"
Sep 18 14:39:10.053418 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:10.053376    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/55cffdc7-1e6c-4cfe-a016-2eb0df1a37d3-lib-modules\") pod \"perf-node-gather-daemonset-zmdm6\" (UID: \"55cffdc7-1e6c-4cfe-a016-2eb0df1a37d3\") " pod="openshift-must-gather-9xtsh/perf-node-gather-daemonset-zmdm6"
Sep 18 14:39:10.064397 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:10.064367    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgdbt\" (UniqueName: \"kubernetes.io/projected/55cffdc7-1e6c-4cfe-a016-2eb0df1a37d3-kube-api-access-fgdbt\") pod \"perf-node-gather-daemonset-zmdm6\" (UID: \"55cffdc7-1e6c-4cfe-a016-2eb0df1a37d3\") " pod="openshift-must-gather-9xtsh/perf-node-gather-daemonset-zmdm6"
Sep 18 14:39:10.174978 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:10.174924    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9xtsh/perf-node-gather-daemonset-zmdm6"
Sep 18 14:39:10.449348 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:10.449322    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9xtsh/perf-node-gather-daemonset-zmdm6"]
Sep 18 14:39:10.451678 ip-10-0-143-120 kubenswrapper[2579]: W0918 14:39:10.451655    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod55cffdc7_1e6c_4cfe_a016_2eb0df1a37d3.slice/crio-294259168ef7c2dc5ad2c3107394cedc891469740a7aeb347473ab54143cc053 WatchSource:0}: Error finding container 294259168ef7c2dc5ad2c3107394cedc891469740a7aeb347473ab54143cc053: Status 404 returned error can't find the container with id 294259168ef7c2dc5ad2c3107394cedc891469740a7aeb347473ab54143cc053
Sep 18 14:39:11.257520 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:11.257485    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9xtsh/perf-node-gather-daemonset-zmdm6" event={"ID":"55cffdc7-1e6c-4cfe-a016-2eb0df1a37d3","Type":"ContainerStarted","Data":"814d15870420ea22f81696d816ea5629c7f71740193dd6b596c9d50942a948ae"}
Sep 18 14:39:11.257938 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:11.257563    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9xtsh/perf-node-gather-daemonset-zmdm6" event={"ID":"55cffdc7-1e6c-4cfe-a016-2eb0df1a37d3","Type":"ContainerStarted","Data":"294259168ef7c2dc5ad2c3107394cedc891469740a7aeb347473ab54143cc053"}
Sep 18 14:39:11.257938 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:11.257601    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9xtsh/perf-node-gather-daemonset-zmdm6"
Sep 18 14:39:11.294125 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:11.294067    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9xtsh/perf-node-gather-daemonset-zmdm6" podStartSLOduration=2.294054519 podStartE2EDuration="2.294054519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 14:39:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:39:11.292005337 +0000 UTC m=+2423.180721267" watchObservedRunningTime="2026-09-18 14:39:11.294054519 +0000 UTC m=+2423.182770449"
Sep 18 14:39:13.994909 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:13.994876    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c6gvb_5bedf54a-fdc9-42b4-a3bb-df8621dd58cb/dns/0.log"
Sep 18 14:39:14.054412 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:14.054372    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c6gvb_5bedf54a-fdc9-42b4-a3bb-df8621dd58cb/kube-rbac-proxy/0.log"
Sep 18 14:39:14.336817 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:14.336783    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t9ks5_690b430b-1984-4119-beae-85786c10b267/dns-node-resolver/0.log"
Sep 18 14:39:14.972565 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:14.972540    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-549d599756-4kgjb_ccb6a3bc-51ff-46d8-8a67-a97bd76f618c/registry/0.log"
Sep 18 14:39:15.046606 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:15.046586    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8n79p_dbb463f3-8dcd-413d-859f-5dcf87c59115/node-ca/0.log"
Sep 18 14:39:16.928601 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:16.928565    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-vklds_e4574081-5495-4cd9-8c39-0c7cded44228/discovery/0.log"
Sep 18 14:39:17.067459 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:17.067437    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86dc75d4b5-ls4xg_6846cc5d-9e5f-4a5a-abbb-54c7163c1f96/kube-auth-proxy/0.log"
Sep 18 14:39:17.270608 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:17.270550    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9xtsh/perf-node-gather-daemonset-zmdm6"
Sep 18 14:39:18.785852 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:18.785781    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gjcx2_7903d08d-219b-453f-9a68-3710e004f0e4/serve-healthcheck-canary/0.log"
Sep 18 14:39:19.564258 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:19.564228    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-52cv4_5ac8c4ef-13cb-43f0-b07b-5540874267d3/kube-rbac-proxy/0.log"
Sep 18 14:39:19.590534 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:19.590494    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-52cv4_5ac8c4ef-13cb-43f0-b07b-5540874267d3/exporter/0.log"
Sep 18 14:39:19.622549 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:19.622483    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-52cv4_5ac8c4ef-13cb-43f0-b07b-5540874267d3/extractor/0.log"
Sep 18 14:39:19.955841 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:19.955783    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-vklds_e4574081-5495-4cd9-8c39-0c7cded44228/discovery/0.log"
Sep 18 14:39:19.993074 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:19.993050    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86dc75d4b5-ls4xg_6846cc5d-9e5f-4a5a-abbb-54c7163c1f96/kube-auth-proxy/0.log"
Sep 18 14:39:21.984253 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:21.984226    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-vklds_e4574081-5495-4cd9-8c39-0c7cded44228/discovery/0.log"
Sep 18 14:39:22.021104 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:22.021080    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86dc75d4b5-ls4xg_6846cc5d-9e5f-4a5a-abbb-54c7163c1f96/kube-auth-proxy/0.log"
Sep 18 14:39:22.542891 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:22.542864    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-l89sx_413479c5-b502-4549-8b20-852bc9d3648f/manager/0.log"
Sep 18 14:39:22.590732 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:22.590710    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-765845fd55-b6jmw_2ef9b7d3-add6-4a4c-b3f3-9da526a53801/manager/0.log"
Sep 18 14:39:22.745784 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:22.745760    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-74d96fdcf9-6h2ct_e3ab8f69-7fd2-46e0-94fb-98db7182cde6/manager/0.log"
Sep 18 14:39:22.787951 ip-10-0-143-120 kubenswrapper[2579]: I0918 14:39:22.787932    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-74d96fdcf9-6h2ct_e3ab8f69-7fd2-46e0-94fb-98db7182cde6/copy-manifests/0.log"