Aug 02 09:31:16.009827 ip-10-0-141-232 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Aug 02 09:31:16.009838 ip-10-0-141-232 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Aug 02 09:31:16.009847 ip-10-0-141-232 systemd[1]: kubelet.service: Failed with result 'resources'.
Aug 02 09:31:16.010224 ip-10-0-141-232 systemd[1]: Failed to start Kubernetes Kubelet.
Aug 02 09:31:26.145212 ip-10-0-141-232 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Aug 02 09:31:26.145232 ip-10-0-141-232 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot a40ad43150314003855cc1e64817cda8 --
Aug 02 09:32:56.159479 ip-10-0-141-232 systemd[1]: Starting Kubernetes Kubelet...
Aug 02 09:32:56.599878 ip-10-0-141-232 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 02 09:32:56.599878 ip-10-0-141-232 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Aug 02 09:32:56.599878 ip-10-0-141-232 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 02 09:32:56.599878 ip-10-0-141-232 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Aug 02 09:32:56.599878 ip-10-0-141-232 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 02 09:32:56.602902 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.602817    2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Aug 02 09:32:56.607869 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.607852    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 02 09:32:56.607869 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.607867    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 02 09:32:56.607869 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.607873    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 02 09:32:56.608051 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.607878    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 02 09:32:56.608051 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.607883    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 02 09:32:56.608051 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.607888    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 02 09:32:56.608051 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.607892    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 02 09:32:56.608051 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.607895    2564 feature_gate.go:328] unrecognized feature gate: Example2
Aug 02 09:32:56.608051 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.607899    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 02 09:32:56.608051 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.607903    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 02 09:32:56.608051 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.607907    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 02 09:32:56.608051 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.607911    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 02 09:32:56.608051 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.607914    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 02 09:32:56.608051 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.607918    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 02 09:32:56.608051 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.607922    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 02 09:32:56.608051 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.607926    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 02 09:32:56.608051 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.607930    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 02 09:32:56.608051 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.607935    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 02 09:32:56.608051 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.607938    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 02 09:32:56.608051 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.607944    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 02 09:32:56.608051 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.607950    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 02 09:32:56.608051 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.607954    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 02 09:32:56.608843 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.607958    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 02 09:32:56.608843 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.607963    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 02 09:32:56.608843 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.607967    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 02 09:32:56.608843 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.607985    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 02 09:32:56.608843 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.607991    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 02 09:32:56.608843 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.607996    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 02 09:32:56.608843 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608001    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 02 09:32:56.608843 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608005    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 02 09:32:56.608843 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608010    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 02 09:32:56.608843 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608014    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 02 09:32:56.608843 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608019    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 02 09:32:56.608843 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608023    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 02 09:32:56.608843 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608027    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 02 09:32:56.608843 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608032    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 02 09:32:56.608843 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608037    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 02 09:32:56.608843 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608042    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 02 09:32:56.608843 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608046    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 02 09:32:56.608843 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608050    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 02 09:32:56.608843 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608054    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 02 09:32:56.608843 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608059    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 02 09:32:56.609357 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608065    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 02 09:32:56.609357 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608069    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 02 09:32:56.609357 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608073    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 02 09:32:56.609357 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608078    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 02 09:32:56.609357 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608082    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 02 09:32:56.609357 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608086    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 02 09:32:56.609357 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608090    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 02 09:32:56.609357 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608094    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 02 09:32:56.609357 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608099    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 02 09:32:56.609357 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608103    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 02 09:32:56.609357 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608107    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 02 09:32:56.609357 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608111    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 02 09:32:56.609357 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608115    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 02 09:32:56.609357 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608120    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 02 09:32:56.609357 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608124    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 02 09:32:56.609357 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608128    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 02 09:32:56.609357 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608132    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 02 09:32:56.609357 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608136    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 02 09:32:56.609357 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608140    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 02 09:32:56.609357 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608144    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 02 09:32:56.609919 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608149    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 02 09:32:56.609919 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608153    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 02 09:32:56.609919 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608157    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 02 09:32:56.609919 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608161    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 02 09:32:56.609919 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608165    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 02 09:32:56.609919 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608172    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 02 09:32:56.609919 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608177    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 02 09:32:56.609919 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608181    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 02 09:32:56.609919 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608186    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 02 09:32:56.609919 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608190    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 02 09:32:56.609919 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608194    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 02 09:32:56.609919 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608198    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 02 09:32:56.609919 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608202    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 02 09:32:56.609919 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608207    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 02 09:32:56.609919 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608211    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 02 09:32:56.609919 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608215    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 02 09:32:56.609919 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608220    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 02 09:32:56.609919 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608224    2564 feature_gate.go:328] unrecognized feature gate: Example
Aug 02 09:32:56.609919 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608244    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 02 09:32:56.610426 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608251    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 02 09:32:56.610426 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608257    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 02 09:32:56.610426 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608261    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 02 09:32:56.610426 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608266    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 02 09:32:56.610426 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608270    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 02 09:32:56.610426 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608274    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 02 09:32:56.610426 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608278    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 02 09:32:56.610426 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608283    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 02 09:32:56.610426 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608288    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 02 09:32:56.610426 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608292    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 02 09:32:56.610426 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608296    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 02 09:32:56.610426 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608300    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 02 09:32:56.610426 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608304    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 02 09:32:56.610426 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608309    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 02 09:32:56.610426 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608313    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 02 09:32:56.610426 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608317    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 02 09:32:56.610426 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608321    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 02 09:32:56.610426 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.608325    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 02 09:32:56.611179 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611164    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 02 09:32:56.611179 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611180    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 02 09:32:56.611301 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611185    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 02 09:32:56.611301 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611189    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 02 09:32:56.611301 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611193    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 02 09:32:56.611301 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611197    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 02 09:32:56.611301 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611202    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 02 09:32:56.611301 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611206    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 02 09:32:56.611301 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611210    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 02 09:32:56.611301 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611215    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 02 09:32:56.611301 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611219    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 02 09:32:56.611301 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611223    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 02 09:32:56.611301 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611227    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 02 09:32:56.611301 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611252    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 02 09:32:56.611301 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611256    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 02 09:32:56.611301 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611261    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 02 09:32:56.611301 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611265    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 02 09:32:56.611301 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611269    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 02 09:32:56.611301 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611273    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 02 09:32:56.611301 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611277    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 02 09:32:56.611301 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611281    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 02 09:32:56.611301 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611284    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 02 09:32:56.612151 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611288    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 02 09:32:56.612151 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611293    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 02 09:32:56.612151 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611297    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 02 09:32:56.612151 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611300    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 02 09:32:56.612151 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611304    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 02 09:32:56.612151 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611308    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 02 09:32:56.612151 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611313    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 02 09:32:56.612151 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611317    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 02 09:32:56.612151 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611321    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 02 09:32:56.612151 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611324    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 02 09:32:56.612151 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611329    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 02 09:32:56.612151 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611333    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 02 09:32:56.612151 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611337    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 02 09:32:56.612151 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611341    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 02 09:32:56.612151 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611345    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 02 09:32:56.612151 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611349    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 02 09:32:56.612151 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611353    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 02 09:32:56.612151 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611357    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 02 09:32:56.612151 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611361    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 02 09:32:56.612151 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611364    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 02 09:32:56.613050 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611370    2564 feature_gate.go:328] unrecognized feature gate: Example2
Aug 02 09:32:56.613050 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611375    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 02 09:32:56.613050 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611379    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 02 09:32:56.613050 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611383    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 02 09:32:56.613050 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611386    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 02 09:32:56.613050 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611390    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 02 09:32:56.613050 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611394    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 02 09:32:56.613050 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611398    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 02 09:32:56.613050 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611403    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 02 09:32:56.613050 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611406    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 02 09:32:56.613050 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611414    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 02 09:32:56.613050 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611420    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 02 09:32:56.613050 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611425    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 02 09:32:56.613050 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611430    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 02 09:32:56.613050 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611435    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 02 09:32:56.613050 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611442    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 02 09:32:56.613050 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611449    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 02 09:32:56.613050 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611456    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 02 09:32:56.613854 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611463    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 02 09:32:56.613854 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611467    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 02 09:32:56.613854 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611471    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 02 09:32:56.613854 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611475    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 02 09:32:56.613854 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611480    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 02 09:32:56.613854 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611485    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 02 09:32:56.613854 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611489    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 02 09:32:56.613854 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611493    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 02 09:32:56.613854 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611497    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 02 09:32:56.613854 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611501    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 02 09:32:56.613854 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611505    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 02 09:32:56.613854 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611509    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 02 09:32:56.613854 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611513    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 02 09:32:56.613854 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611517    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 02 09:32:56.613854 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611522    2564 feature_gate.go:328] unrecognized feature gate: Example
Aug 02 09:32:56.613854 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611526    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 02 09:32:56.613854 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611530    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 02 09:32:56.613854 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611534    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 02 09:32:56.613854 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611538    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 02 09:32:56.613854 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611542    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 02 09:32:56.614601 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611546    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 02 09:32:56.614601 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611550    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 02 09:32:56.614601 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611554    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 02 09:32:56.614601 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611559    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 02 09:32:56.614601 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611563    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 02 09:32:56.614601 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611567    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 02 09:32:56.614601 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611571    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 02 09:32:56.614601 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611576    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 02 09:32:56.614601 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611580    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 02 09:32:56.614601 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611584    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 02 09:32:56.614601 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611588    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 02 09:32:56.614601 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611592    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 02 09:32:56.614601 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611596    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 02 09:32:56.614601 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611600    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 02 09:32:56.614601 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611604    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 02 09:32:56.614601 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611608    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 02 09:32:56.614601 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611615    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 02 09:32:56.614601 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611619    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 02 09:32:56.614601 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.611623    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 02 09:32:56.614601 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612189    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 02 09:32:56.615088 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612197    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 02 09:32:56.615088 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612202    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 02 09:32:56.615088 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612206    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 02 09:32:56.615088 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612210    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 02 09:32:56.615088 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612214    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 02 09:32:56.615088 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612219    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 02 09:32:56.615088 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612223    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 02 09:32:56.615088 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612247    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 02 09:32:56.615088 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612252    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 02 09:32:56.615088 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612256    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 02 09:32:56.615088 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612261    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 02 09:32:56.615088 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612265    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 02 09:32:56.615088 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612270    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 02 09:32:56.615088 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612274    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 02 09:32:56.615088 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612278    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 02 09:32:56.615088 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612285    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 02 09:32:56.615088 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612290    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 02 09:32:56.615088 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612296    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 02 09:32:56.615088 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612300    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 02 09:32:56.615088 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612305    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 02 09:32:56.615696 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612310    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 02 09:32:56.615696 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612314    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 02 09:32:56.615696 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612318    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 02 09:32:56.615696 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612323    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 02 09:32:56.615696 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612329    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 02 09:32:56.615696 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612335    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 02 09:32:56.615696 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612339    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 02 09:32:56.615696 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612343    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 02 09:32:56.615696 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612349    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 02 09:32:56.615696 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612353    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 02 09:32:56.615696 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612357    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 02 09:32:56.615696 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612361    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 02 09:32:56.615696 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612365    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 02 09:32:56.615696 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612369    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 02 09:32:56.615696 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612373    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 02 09:32:56.615696 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612378    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 02 09:32:56.615696 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612382    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 02 09:32:56.615696 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612387    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 02 09:32:56.615696 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612391    2564 feature_gate.go:328] unrecognized feature gate: Example2
Aug 02 09:32:56.615696 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612396    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 02 09:32:56.616380 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612401    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 02 09:32:56.616380 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612405    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 02 09:32:56.616380 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612409    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 02 09:32:56.616380 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612413    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 02 09:32:56.616380 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612417    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 02 09:32:56.616380 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612421    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 02 09:32:56.616380 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612425    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 02 09:32:56.616380 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612429    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 02 09:32:56.616380 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612433    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 02 09:32:56.616380 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612437    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 02 09:32:56.616380 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612442    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 02 09:32:56.616380 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612446    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 02 09:32:56.616380 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612450    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 02 09:32:56.616380 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612454    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 02 09:32:56.616380 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612458    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 02 09:32:56.616380 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612462    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 02 09:32:56.616380 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612469    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 02 09:32:56.616380 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612474    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 02 09:32:56.616380 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612479    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 02 09:32:56.616935 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612483    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 02 09:32:56.616935 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612489    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 02 09:32:56.616935 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612493    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 02 09:32:56.616935 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612497    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 02 09:32:56.616935 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612502    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 02 09:32:56.616935 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612506    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 02 09:32:56.616935 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612510    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 02 09:32:56.616935 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612514    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 02 09:32:56.616935 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612518    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 02 09:32:56.616935 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612522    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 02 09:32:56.616935 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612526    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 02 09:32:56.616935 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612530    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 02 09:32:56.616935 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612534    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 02 09:32:56.616935 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612539    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 02 09:32:56.616935 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612543    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 02 09:32:56.616935 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612548    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 02 09:32:56.616935 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612552    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 02 09:32:56.616935 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612555    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 02 09:32:56.616935 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612560    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 02 09:32:56.616935 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612564    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 02 09:32:56.617484 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612568    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 02 09:32:56.617484 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612572    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 02 09:32:56.617484 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612576    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 02 09:32:56.617484 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612580    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 02 09:32:56.617484 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612584    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 02 09:32:56.617484 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612589    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 02 09:32:56.617484 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612593    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 02 09:32:56.617484 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612597    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 02 09:32:56.617484 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612601    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 02 09:32:56.617484 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612605    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 02 09:32:56.617484 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612609    2564 feature_gate.go:328] unrecognized feature gate: Example
Aug 02 09:32:56.617484 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612614    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 02 09:32:56.617484 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612618    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 02 09:32:56.617484 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612622    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 02 09:32:56.617484 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612626    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 02 09:32:56.617484 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612630    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 02 09:32:56.617484 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612636    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 02 09:32:56.617484 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612640    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 02 09:32:56.617484 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.612644    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 02 09:32:56.617484 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.612734    2564 flags.go:64] FLAG: --address="0.0.0.0"
Aug 02 09:32:56.618058 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.612744    2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Aug 02 09:32:56.618058 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.612752    2564 flags.go:64] FLAG: --anonymous-auth="true"
Aug 02 09:32:56.618058 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.612759    2564 flags.go:64] FLAG: --application-metrics-count-limit="100"
Aug 02 09:32:56.618058 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.612765    2564 flags.go:64] FLAG: --authentication-token-webhook="false"
Aug 02 09:32:56.618058 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.612771    2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Aug 02 09:32:56.618058 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.612777    2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Aug 02 09:32:56.618058 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.612784    2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Aug 02 09:32:56.618058 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.612790    2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Aug 02 09:32:56.618058 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.612794    2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Aug 02 09:32:56.618058 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.612800    2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Aug 02 09:32:56.618058 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.612805    2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Aug 02 09:32:56.618058 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.612810    2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Aug 02 09:32:56.618058 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.612815    2564 flags.go:64] FLAG: --cgroup-root=""
Aug 02 09:32:56.618058 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.612820    2564 flags.go:64] FLAG: --cgroups-per-qos="true"
Aug 02 09:32:56.618058 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.612825    2564 flags.go:64] FLAG: --client-ca-file=""
Aug 02 09:32:56.618058 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.612829    2564 flags.go:64] FLAG: --cloud-provider="external"
Aug 02 09:32:56.618058 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.612836    2564 flags.go:64] FLAG: --cluster-dns="[]"
Aug 02 09:32:56.618058 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.612843    2564 flags.go:64] FLAG: --cluster-domain=""
Aug 02 09:32:56.618058 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.612848    2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Aug 02 09:32:56.618058 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.612853    2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Aug 02 09:32:56.618058 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.612858    2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Aug 02 09:32:56.618058 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.612863    2564 flags.go:64] FLAG: --container-log-max-files="5"
Aug 02 09:32:56.618058 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.612869    2564 flags.go:64] FLAG: --container-log-max-size="10Mi"
Aug 02 09:32:56.618058 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.612874    2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Aug 02 09:32:56.618696 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.612878    2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Aug 02 09:32:56.618696 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.612884    2564 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Aug 02 09:32:56.618696 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.612888    2564 flags.go:64] FLAG: --contention-profiling="false"
Aug 02 09:32:56.618696 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.612894    2564 flags.go:64] FLAG: --cpu-cfs-quota="true"
Aug 02 09:32:56.618696 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.612898    2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Aug 02 09:32:56.618696 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.612903    2564 flags.go:64] FLAG: --cpu-manager-policy="none"
Aug 02 09:32:56.618696 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.612908    2564 flags.go:64] FLAG: --cpu-manager-policy-options=""
Aug 02 09:32:56.618696 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.612914    2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Aug 02 09:32:56.618696 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.612918    2564 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Aug 02 09:32:56.618696 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.612923    2564 flags.go:64] FLAG: --enable-debugging-handlers="true"
Aug 02 09:32:56.618696 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.612927    2564 flags.go:64] FLAG: --enable-load-reader="false"
Aug 02 09:32:56.618696 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.612932    2564 flags.go:64] FLAG: --enable-server="true"
Aug 02 09:32:56.618696 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.612937    2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Aug 02 09:32:56.618696 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.612943    2564 flags.go:64] FLAG: --event-burst="100"
Aug 02 09:32:56.618696 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.612948    2564 flags.go:64] FLAG: --event-qps="50"
Aug 02 09:32:56.618696 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.612953    2564 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Aug 02 09:32:56.618696 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.612958    2564 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Aug 02 09:32:56.618696 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.612963    2564 flags.go:64] FLAG: --eviction-hard=""
Aug 02 09:32:56.618696 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.612969    2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Aug 02 09:32:56.618696 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.612974    2564 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Aug 02 09:32:56.618696 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.612979    2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Aug 02 09:32:56.618696 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.612984    2564 flags.go:64] FLAG: --eviction-soft=""
Aug 02 09:32:56.618696 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.612988    2564 flags.go:64] FLAG: --eviction-soft-grace-period=""
Aug 02 09:32:56.618696 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.612993    2564 flags.go:64] FLAG: --exit-on-lock-contention="false"
Aug 02 09:32:56.618696 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613000    2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Aug 02 09:32:56.619333 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613004    2564 flags.go:64] FLAG: --experimental-mounter-path=""
Aug 02 09:32:56.619333 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613009    2564 flags.go:64] FLAG: --fail-cgroupv1="false"
Aug 02 09:32:56.619333 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613013    2564 flags.go:64] FLAG: --fail-swap-on="true"
Aug 02 09:32:56.619333 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613018    2564 flags.go:64] FLAG: --feature-gates=""
Aug 02 09:32:56.619333 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613024    2564 flags.go:64] FLAG: --file-check-frequency="20s"
Aug 02 09:32:56.619333 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613029    2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Aug 02 09:32:56.619333 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613035    2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Aug 02 09:32:56.619333 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613041    2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Aug 02 09:32:56.619333 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613045    2564 flags.go:64] FLAG: --healthz-port="10248"
Aug 02 09:32:56.619333 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613050    2564 flags.go:64] FLAG: --help="false"
Aug 02 09:32:56.619333 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613054    2564 flags.go:64] FLAG: --hostname-override="ip-10-0-141-232.ec2.internal"
Aug 02 09:32:56.619333 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613060    2564 flags.go:64] FLAG: --housekeeping-interval="10s"
Aug 02 09:32:56.619333 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613064    2564 flags.go:64] FLAG: --http-check-frequency="20s"
Aug 02 09:32:56.619333 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613069    2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Aug 02 09:32:56.619333 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613074    2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Aug 02 09:32:56.619333 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613080    2564 flags.go:64] FLAG: --image-gc-high-threshold="85"
Aug 02 09:32:56.619333 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613085    2564 flags.go:64] FLAG: --image-gc-low-threshold="80"
Aug 02 09:32:56.619333 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613089    2564 flags.go:64] FLAG: --image-service-endpoint=""
Aug 02 09:32:56.619333 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613094    2564 flags.go:64] FLAG: --kernel-memcg-notification="false"
Aug 02 09:32:56.619333 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613098    2564 flags.go:64] FLAG: --kube-api-burst="100"
Aug 02 09:32:56.619333 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613103    2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Aug 02 09:32:56.619333 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613108    2564 flags.go:64] FLAG: --kube-api-qps="50"
Aug 02 09:32:56.619333 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613112    2564 flags.go:64] FLAG: --kube-reserved=""
Aug 02 09:32:56.619333 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613117    2564 flags.go:64] FLAG: --kube-reserved-cgroup=""
Aug 02 09:32:56.619924 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613122    2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Aug 02 09:32:56.619924 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613128    2564 flags.go:64] FLAG: --kubelet-cgroups=""
Aug 02 09:32:56.619924 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613132    2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Aug 02 09:32:56.619924 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613137    2564 flags.go:64] FLAG: --lock-file=""
Aug 02 09:32:56.619924 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613141    2564 flags.go:64] FLAG: --log-cadvisor-usage="false"
Aug 02 09:32:56.619924 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613146    2564 flags.go:64] FLAG: --log-flush-frequency="5s"
Aug 02 09:32:56.619924 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613151    2564 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Aug 02 09:32:56.619924 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613161    2564 flags.go:64] FLAG: --log-json-split-stream="false"
Aug 02 09:32:56.619924 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613166    2564 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Aug 02 09:32:56.619924 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613170    2564 flags.go:64] FLAG: --log-text-split-stream="false"
Aug 02 09:32:56.619924 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613175    2564 flags.go:64] FLAG: --logging-format="text"
Aug 02 09:32:56.619924 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613180    2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Aug 02 09:32:56.619924 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613185    2564 flags.go:64] FLAG: --make-iptables-util-chains="true"
Aug 02 09:32:56.619924 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613189    2564 flags.go:64] FLAG: --manifest-url=""
Aug 02 09:32:56.619924 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613195    2564 flags.go:64] FLAG: --manifest-url-header=""
Aug 02 09:32:56.619924 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613202    2564 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Aug 02 09:32:56.619924 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613207    2564 flags.go:64] FLAG: --max-open-files="1000000"
Aug 02 09:32:56.619924 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613213    2564 flags.go:64] FLAG: --max-pods="110"
Aug 02 09:32:56.619924 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613218    2564 flags.go:64] FLAG: --maximum-dead-containers="-1"
Aug 02 09:32:56.619924 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613222    2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Aug 02 09:32:56.619924 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613227    2564 flags.go:64] FLAG: --memory-manager-policy="None"
Aug 02 09:32:56.619924 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613249    2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Aug 02 09:32:56.619924 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613255    2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Aug 02 09:32:56.619924 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613259    2564 flags.go:64] FLAG: --node-ip="0.0.0.0"
Aug 02 09:32:56.619924 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613264    2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Aug 02 09:32:56.620601 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613275    2564 flags.go:64] FLAG: --node-status-max-images="50"
Aug 02 09:32:56.620601 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613280    2564 flags.go:64] FLAG: --node-status-update-frequency="10s"
Aug 02 09:32:56.620601 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613285    2564 flags.go:64] FLAG: --oom-score-adj="-999"
Aug 02 09:32:56.620601 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613290    2564 flags.go:64] FLAG: --pod-cidr=""
Aug 02 09:32:56.620601 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613294    2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8"
Aug 02 09:32:56.620601 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613302    2564 flags.go:64] FLAG: --pod-manifest-path=""
Aug 02 09:32:56.620601 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613307    2564 flags.go:64] FLAG: --pod-max-pids="-1"
Aug 02 09:32:56.620601 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613312    2564 flags.go:64] FLAG: --pods-per-core="0"
Aug 02 09:32:56.620601 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613318    2564 flags.go:64] FLAG: --port="10250"
Aug 02 09:32:56.620601 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613323    2564 flags.go:64] FLAG: --protect-kernel-defaults="false"
Aug 02 09:32:56.620601 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613327    2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08d96bf05036458b1"
Aug 02 09:32:56.620601 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613333    2564 flags.go:64] FLAG: --qos-reserved=""
Aug 02 09:32:56.620601 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613338    2564 flags.go:64] FLAG: --read-only-port="10255"
Aug 02 09:32:56.620601 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613343    2564 flags.go:64] FLAG: --register-node="true"
Aug 02 09:32:56.620601 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613347    2564 flags.go:64] FLAG: --register-with-taints=""
Aug 02 09:32:56.620601 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613354    2564 flags.go:64] FLAG: --registry-burst="10"
Aug 02 09:32:56.620601 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613359    2564 flags.go:64] FLAG: --registry-qps="5"
Aug 02 09:32:56.620601 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613364    2564 flags.go:64] FLAG: --reserved-cpus=""
Aug 02 09:32:56.620601 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613368    2564 flags.go:64] FLAG: --reserved-memory=""
Aug 02 09:32:56.620601 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613374    2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Aug 02 09:32:56.620601 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613379    2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Aug 02 09:32:56.620601 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613385    2564 flags.go:64] FLAG: --rotate-certificates="false"
Aug 02 09:32:56.620601 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613389    2564 flags.go:64] FLAG: --rotate-server-certificates="false"
Aug 02 09:32:56.620601 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613394    2564 flags.go:64] FLAG: --runonce="false"
Aug 02 09:32:56.620601 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613399    2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Aug 02 09:32:56.621304 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613404    2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Aug 02 09:32:56.621304 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613409    2564 flags.go:64] FLAG: --seccomp-default="false"
Aug 02 09:32:56.621304 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613414    2564 flags.go:64] FLAG: --serialize-image-pulls="true"
Aug 02 09:32:56.621304 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613419    2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Aug 02 09:32:56.621304 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613424    2564 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Aug 02 09:32:56.621304 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613429    2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Aug 02 09:32:56.621304 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613434    2564 flags.go:64] FLAG: --storage-driver-password="root"
Aug 02 09:32:56.621304 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613438    2564 flags.go:64] FLAG: --storage-driver-secure="false"
Aug 02 09:32:56.621304 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613443    2564 flags.go:64] FLAG: --storage-driver-table="stats"
Aug 02 09:32:56.621304 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613448    2564 flags.go:64] FLAG: --storage-driver-user="root"
Aug 02 09:32:56.621304 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613452    2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Aug 02 09:32:56.621304 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613457    2564 flags.go:64] FLAG: --sync-frequency="1m0s"
Aug 02 09:32:56.621304 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613462    2564 flags.go:64] FLAG: --system-cgroups=""
Aug 02 09:32:56.621304 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613466    2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Aug 02 09:32:56.621304 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613475    2564 flags.go:64] FLAG: --system-reserved-cgroup=""
Aug 02 09:32:56.621304 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613480    2564 flags.go:64] FLAG: --tls-cert-file=""
Aug 02 09:32:56.621304 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613486    2564 flags.go:64] FLAG: --tls-cipher-suites="[]"
Aug 02 09:32:56.621304 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613492    2564 flags.go:64] FLAG: --tls-min-version=""
Aug 02 09:32:56.621304 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613497    2564 flags.go:64] FLAG: --tls-private-key-file=""
Aug 02 09:32:56.621304 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613501    2564 flags.go:64] FLAG: --topology-manager-policy="none"
Aug 02 09:32:56.621304 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613506    2564 flags.go:64] FLAG: --topology-manager-policy-options=""
Aug 02 09:32:56.621304 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613510    2564 flags.go:64] FLAG: --topology-manager-scope="container"
Aug 02 09:32:56.621304 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613516    2564 flags.go:64] FLAG: --v="2"
Aug 02 09:32:56.621304 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613522    2564 flags.go:64] FLAG: --version="false"
Aug 02 09:32:56.621304 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613528    2564 flags.go:64] FLAG: --vmodule=""
Aug 02 09:32:56.621947 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613534    2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Aug 02 09:32:56.621947 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.613539    2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Aug 02 09:32:56.621947 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613707    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 02 09:32:56.621947 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613716    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 02 09:32:56.621947 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613721    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 02 09:32:56.621947 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613726    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 02 09:32:56.621947 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613730    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 02 09:32:56.621947 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613734    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 02 09:32:56.621947 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613738    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 02 09:32:56.621947 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613743    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 02 09:32:56.621947 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613750    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 02 09:32:56.621947 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613756    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 02 09:32:56.621947 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613760    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 02 09:32:56.621947 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613765    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 02 09:32:56.621947 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613770    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 02 09:32:56.621947 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613774    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 02 09:32:56.621947 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613780    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 02 09:32:56.621947 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613786    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 02 09:32:56.621947 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613790    2564 feature_gate.go:328] unrecognized feature gate: Example
Aug 02 09:32:56.622456 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613794    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 02 09:32:56.622456 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613799    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 02 09:32:56.622456 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613803    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 02 09:32:56.622456 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613807    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 02 09:32:56.622456 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613811    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 02 09:32:56.622456 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613816    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 02 09:32:56.622456 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613821    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 02 09:32:56.622456 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613827    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 02 09:32:56.622456 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613833    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 02 09:32:56.622456 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613838    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 02 09:32:56.622456 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613844    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 02 09:32:56.622456 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613849    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 02 09:32:56.622456 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613853    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 02 09:32:56.622456 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613857    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 02 09:32:56.622456 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613862    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 02 09:32:56.622456 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613866    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 02 09:32:56.622456 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613872    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 02 09:32:56.622456 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613876    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 02 09:32:56.622456 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613880    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 02 09:32:56.622930 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613884    2564 feature_gate.go:328] unrecognized feature gate: Example2
Aug 02 09:32:56.622930 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613888    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 02 09:32:56.622930 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613893    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 02 09:32:56.622930 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613897    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 02 09:32:56.622930 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613901    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 02 09:32:56.622930 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613906    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 02 09:32:56.622930 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613910    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 02 09:32:56.622930 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613914    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 02 09:32:56.622930 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613918    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 02 09:32:56.622930 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613922    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 02 09:32:56.622930 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613927    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 02 09:32:56.622930 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613931    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 02 09:32:56.622930 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613935    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 02 09:32:56.622930 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613939    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 02 09:32:56.622930 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613944    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 02 09:32:56.622930 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613948    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 02 09:32:56.622930 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613952    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 02 09:32:56.622930 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613956    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 02 09:32:56.622930 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613960    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 02 09:32:56.622930 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613965    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 02 09:32:56.623444 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613970    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 02 09:32:56.623444 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613974    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 02 09:32:56.623444 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613978    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 02 09:32:56.623444 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613984    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 02 09:32:56.623444 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613988    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 02 09:32:56.623444 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613992    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 02 09:32:56.623444 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.613996    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 02 09:32:56.623444 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.614000    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 02 09:32:56.623444 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.614004    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 02 09:32:56.623444 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.614010    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 02 09:32:56.623444 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.614014    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 02 09:32:56.623444 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.614019    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 02 09:32:56.623444 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.614023    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 02 09:32:56.623444 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.614027    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 02 09:32:56.623444 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.614032    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 02 09:32:56.623444 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.614036    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 02 09:32:56.623444 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.614040    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 02 09:32:56.623444 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.614044    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 02 09:32:56.623444 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.614048    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 02 09:32:56.623444 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.614052    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 02 09:32:56.623935 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.614056    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 02 09:32:56.623935 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.614061    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 02 09:32:56.623935 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.614065    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 02 09:32:56.623935 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.614069    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 02 09:32:56.623935 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.614073    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 02 09:32:56.623935 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.614077    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 02 09:32:56.623935 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.614081    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 02 09:32:56.623935 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.614085    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 02 09:32:56.623935 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.614089    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 02 09:32:56.623935 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.614094    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 02 09:32:56.623935 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.614098    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 02 09:32:56.623935 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.614102    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 02 09:32:56.623935 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.614107    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 02 09:32:56.623935 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.614111    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 02 09:32:56.623935 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.614116    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 02 09:32:56.623935 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.614121    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 02 09:32:56.623935 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.614126    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 02 09:32:56.623935 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.614130    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 02 09:32:56.623935 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.614134    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 02 09:32:56.623935 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.614138    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 02 09:32:56.624444 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.614143    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 02 09:32:56.624444 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.614149    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 02 09:32:56.624444 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.614153    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 02 09:32:56.624444 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.615212    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Aug 02 09:32:56.624444 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.622257    2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Aug 02 09:32:56.624444 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.622270    2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Aug 02 09:32:56.624444 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.622289    2564 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Aug 02 09:32:56.624444 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.622295    2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Aug 02 09:32:56.624444 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622354    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 02 09:32:56.624444 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622360    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 02 09:32:56.624444 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622364    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 02 09:32:56.624444 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622367    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 02 09:32:56.624444 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622370    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 02 09:32:56.624444 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622373    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 02 09:32:56.624444 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622376    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 02 09:32:56.624823 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622379    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 02 09:32:56.624823 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622382    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 02 09:32:56.624823 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622384    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 02 09:32:56.624823 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622387    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 02 09:32:56.624823 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622390    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 02 09:32:56.624823 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622393    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 02 09:32:56.624823 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622395    2564 feature_gate.go:328] unrecognized feature gate: Example
Aug 02 09:32:56.624823 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622399    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 02 09:32:56.624823 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622402    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 02 09:32:56.624823 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622405    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 02 09:32:56.624823 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622408    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 02 09:32:56.624823 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622410    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 02 09:32:56.624823 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622413    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 02 09:32:56.624823 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622416    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 02 09:32:56.624823 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622418    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 02 09:32:56.624823 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622421    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 02 09:32:56.624823 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622423    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 02 09:32:56.624823 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622426    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 02 09:32:56.624823 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622428    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 02 09:32:56.624823 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622431    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 02 09:32:56.625357 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622434    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 02 09:32:56.625357 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622436    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 02 09:32:56.625357 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622439    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 02 09:32:56.625357 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622442    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 02 09:32:56.625357 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622445    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 02 09:32:56.625357 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622447    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 02 09:32:56.625357 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622450    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 02 09:32:56.625357 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622453    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 02 09:32:56.625357 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622455    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 02 09:32:56.625357 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622458    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 02 09:32:56.625357 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622461    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 02 09:32:56.625357 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622464    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 02 09:32:56.625357 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622467    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 02 09:32:56.625357 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622471    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 02 09:32:56.625357 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622475    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 02 09:32:56.625357 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622478    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 02 09:32:56.625357 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622481    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 02 09:32:56.625357 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622484    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 02 09:32:56.625357 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622487    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 02 09:32:56.625357 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622489    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 02 09:32:56.625848 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622492    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 02 09:32:56.625848 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622496    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 02 09:32:56.625848 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622499    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 02 09:32:56.625848 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622503    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 02 09:32:56.625848 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622505    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 02 09:32:56.625848 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622508    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 02 09:32:56.625848 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622511    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 02 09:32:56.625848 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622514    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 02 09:32:56.625848 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622517    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 02 09:32:56.625848 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622520    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 02 09:32:56.625848 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622522    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 02 09:32:56.625848 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622525    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 02 09:32:56.625848 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622528    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 02 09:32:56.625848 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622531    2564 feature_gate.go:328] unrecognized feature gate: Example2
Aug 02 09:32:56.625848 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622533    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 02 09:32:56.625848 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622536    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 02 09:32:56.625848 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622539    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 02 09:32:56.625848 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622541    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 02 09:32:56.625848 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622544    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 02 09:32:56.626374 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622547    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 02 09:32:56.626374 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622549    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 02 09:32:56.626374 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622552    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 02 09:32:56.626374 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622554    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 02 09:32:56.626374 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622557    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 02 09:32:56.626374 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622559    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 02 09:32:56.626374 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622562    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 02 09:32:56.626374 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622564    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 02 09:32:56.626374 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622567    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 02 09:32:56.626374 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622569    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 02 09:32:56.626374 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622572    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 02 09:32:56.626374 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622574    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 02 09:32:56.626374 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622577    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 02 09:32:56.626374 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622579    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 02 09:32:56.626374 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622581    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 02 09:32:56.626374 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622584    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 02 09:32:56.626374 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622587    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 02 09:32:56.626374 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622589    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 02 09:32:56.626374 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622592    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 02 09:32:56.626374 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622594    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 02 09:32:56.626862 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622597    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 02 09:32:56.626862 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622600    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 02 09:32:56.626862 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622602    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 02 09:32:56.626862 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622607    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 02 09:32:56.626862 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622610    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 02 09:32:56.626862 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622613    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 02 09:32:56.626862 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622616    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 02 09:32:56.626862 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622618    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 02 09:32:56.626862 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622621    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 02 09:32:56.626862 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622623    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 02 09:32:56.626862 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622626    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 02 09:32:56.626862 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622629    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 02 09:32:56.626862 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622631    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 02 09:32:56.626862 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.622636    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Aug 02 09:32:56.627207 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622741    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 02 09:32:56.627207 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622745    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 02 09:32:56.627207 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622748    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 02 09:32:56.627207 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622751    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 02 09:32:56.627207 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622754    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 02 09:32:56.627207 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622757    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 02 09:32:56.627207 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622760    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 02 09:32:56.627207 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622763    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 02 09:32:56.627207 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622766    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 02 09:32:56.627207 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622768    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 02 09:32:56.627207 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622771    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 02 09:32:56.627207 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622774    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 02 09:32:56.627207 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622777    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 02 09:32:56.627207 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622780    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 02 09:32:56.627207 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622783    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 02 09:32:56.627207 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622788    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 02 09:32:56.627207 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622791    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 02 09:32:56.627207 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622795    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 02 09:32:56.627207 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622797    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 02 09:32:56.627675 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622801    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 02 09:32:56.627675 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622804    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 02 09:32:56.627675 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622807    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 02 09:32:56.627675 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622809    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 02 09:32:56.627675 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622812    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 02 09:32:56.627675 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622814    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 02 09:32:56.627675 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622817    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 02 09:32:56.627675 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622819    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 02 09:32:56.627675 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622822    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 02 09:32:56.627675 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622824    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 02 09:32:56.627675 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622827    2564 feature_gate.go:328] unrecognized feature gate: Example2
Aug 02 09:32:56.627675 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622830    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 02 09:32:56.627675 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622832    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 02 09:32:56.627675 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622835    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 02 09:32:56.627675 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622839    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 02 09:32:56.627675 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622842    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 02 09:32:56.627675 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622845    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 02 09:32:56.627675 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622848    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 02 09:32:56.627675 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622851    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 02 09:32:56.627675 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622853    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 02 09:32:56.628147 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622856    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 02 09:32:56.628147 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622860    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 02 09:32:56.628147 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622863    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 02 09:32:56.628147 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622866    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 02 09:32:56.628147 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622869    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 02 09:32:56.628147 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622872    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 02 09:32:56.628147 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622874    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 02 09:32:56.628147 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622877    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 02 09:32:56.628147 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622880    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 02 09:32:56.628147 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622883    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 02 09:32:56.628147 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622886    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 02 09:32:56.628147 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622889    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 02 09:32:56.628147 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622892    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 02 09:32:56.628147 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622895    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 02 09:32:56.628147 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622897    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 02 09:32:56.628147 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622900    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 02 09:32:56.628147 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622902    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 02 09:32:56.628147 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622905    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 02 09:32:56.628147 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622907    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 02 09:32:56.628147 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622910    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 02 09:32:56.628670 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622913    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 02 09:32:56.628670 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622916    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 02 09:32:56.628670 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622918    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 02 09:32:56.628670 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622921    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 02 09:32:56.628670 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622923    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 02 09:32:56.628670 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622926    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 02 09:32:56.628670 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622928    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 02 09:32:56.628670 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622931    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 02 09:32:56.628670 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622934    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 02 09:32:56.628670 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622936    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 02 09:32:56.628670 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622939    2564 feature_gate.go:328] unrecognized feature gate: Example
Aug 02 09:32:56.628670 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622942    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 02 09:32:56.628670 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622945    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 02 09:32:56.628670 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622947    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 02 09:32:56.628670 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622950    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 02 09:32:56.628670 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622952    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 02 09:32:56.628670 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622955    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 02 09:32:56.628670 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622957    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 02 09:32:56.628670 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622960    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 02 09:32:56.629124 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622962    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 02 09:32:56.629124 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622980    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 02 09:32:56.629124 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622984    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 02 09:32:56.629124 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622987    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 02 09:32:56.629124 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622990    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 02 09:32:56.629124 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622993    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 02 09:32:56.629124 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622996    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 02 09:32:56.629124 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.622998    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 02 09:32:56.629124 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.623001    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 02 09:32:56.629124 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.623003    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 02 09:32:56.629124 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.623006    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 02 09:32:56.629124 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.623008    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 02 09:32:56.629124 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.623011    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 02 09:32:56.629124 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.623014    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 02 09:32:56.629124 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.623016    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 02 09:32:56.629124 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.623019    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 02 09:32:56.629124 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.623021    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 02 09:32:56.629124 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.623023    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 02 09:32:56.629124 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.623026    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 02 09:32:56.629124 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.623028    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 02 09:32:56.629662 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:56.623031    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 02 09:32:56.629662 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.623035    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Aug 02 09:32:56.629662 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.623122    2564 server.go:962] "Client rotation is on, will bootstrap in background"
Aug 02 09:32:56.629662 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.625442    2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Aug 02 09:32:56.629662 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.627183    2564 server.go:1019] "Starting client certificate rotation"
Aug 02 09:32:56.629662 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.627279    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Aug 02 09:32:56.629662 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.627326    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Aug 02 09:32:56.630344 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.630330    2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Aug 02 09:32:56.635855 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.635836    2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Aug 02 09:32:56.649773 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.649754    2564 log.go:25] "Validated CRI v1 runtime API"
Aug 02 09:32:56.652115 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.652101    2564 log.go:25] "Validated CRI v1 image API"
Aug 02 09:32:56.652885 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.652865    2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Aug 02 09:32:56.655438 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.655418    2564 fs.go:135] Filesystem UUIDs: map[1e24349c-4518-49d5-b5de-4f6625a33f2a:/dev/nvme0n1p4 79380afe-3253-481c-9afc-469cfad7afcb:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Aug 02 09:32:56.655496 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.655438    2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Aug 02 09:32:56.657761 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.657746    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Aug 02 09:32:56.660671 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.660578    2564 manager.go:216] Machine: {Timestamp:2026-08-02 09:32:56.659118093 +0000 UTC m=+0.373923533 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101309 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d8811b4df25812f72a66053add9af SystemUUID:ec2d8811-b4df-2581-2f72-a66053add9af BootID:a40ad431-5031-4003-855c-c1e64817cda8 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f2:f5:a1:9b:ab Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f2:f5:a1:9b:ab Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8e:44:c0:55:87:ba Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Aug 02 09:32:56.660671 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.660669    2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Aug 02 09:32:56.660772 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.660738    2564 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Aug 02 09:32:56.664177 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.664156    2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Aug 02 09:32:56.664317 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.664180    2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-232.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Aug 02 09:32:56.665038 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.665028    2564 topology_manager.go:138] "Creating topology manager with none policy"
Aug 02 09:32:56.665076 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.665040    2564 container_manager_linux.go:309] "Creating device plugin manager"
Aug 02 09:32:56.665076 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.665047    2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Aug 02 09:32:56.665849 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.665840    2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Aug 02 09:32:56.667024 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.667015    2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Aug 02 09:32:56.672116 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.672105    2564 state_mem.go:36] "Initialized new in-memory state store"
Aug 02 09:32:56.672227 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.672219    2564 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Aug 02 09:32:56.672270 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.672266    2564 kubelet.go:486] "Attempting to sync node with API server"
Aug 02 09:32:56.672304 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.672278    2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Aug 02 09:32:56.672335 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.672304    2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Aug 02 09:32:56.672335 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.672315    2564 kubelet.go:398] "Adding apiserver pod source"
Aug 02 09:32:56.672335 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.672327    2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Aug 02 09:32:56.672740 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.672722    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Aug 02 09:32:56.672779 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.672755    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Aug 02 09:32:56.675177 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.675163    2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Aug 02 09:32:56.676961 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.676949    2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Aug 02 09:32:56.677005 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.676979    2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Aug 02 09:32:56.677721 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.677702    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vd76q"
Aug 02 09:32:56.678377 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.678364    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Aug 02 09:32:56.678419 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.678382    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Aug 02 09:32:56.678419 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.678389    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Aug 02 09:32:56.678419 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.678395    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Aug 02 09:32:56.678419 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.678403    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Aug 02 09:32:56.678419 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.678409    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Aug 02 09:32:56.678556 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.678427    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Aug 02 09:32:56.678556 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.678433    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Aug 02 09:32:56.678556 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.678439    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Aug 02 09:32:56.678556 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.678444    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Aug 02 09:32:56.678556 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.678459    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Aug 02 09:32:56.678556 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.678464    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Aug 02 09:32:56.679354 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.679344    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Aug 02 09:32:56.679354 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.679353    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Aug 02 09:32:56.682099 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.682069    2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-232.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Aug 02 09:32:56.682409 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.682398    2564 server.go:1268] "Started kubelet"
Aug 02 09:32:56.682504 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.682479    2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Aug 02 09:32:56.682539 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.682522    2564 server_v1.go:49] "podresources" method="list" useActivePods=true
Aug 02 09:32:56.683108 ip-10-0-141-232 systemd[1]: Started Kubernetes Kubelet.
Aug 02 09:32:56.685819 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:32:56.685788    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-232.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Aug 02 09:32:56.685819 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.685795    2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Aug 02 09:32:56.685940 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:32:56.685824    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Aug 02 09:32:56.688979 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.687028    2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Aug 02 09:32:56.688979 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.687263    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vd76q"
Aug 02 09:32:56.688979 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:32:56.688322    2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Aug 02 09:32:56.689564 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.689550    2564 server.go:310] "Adding debug handlers to kubelet server"
Aug 02 09:32:56.692668 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:32:56.691633    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-232.ec2.internal.18c7f332c6a60b3a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-232.ec2.internal,UID:ip-10-0-141-232.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-232.ec2.internal,},FirstTimestamp:2026-08-02 09:32:56.682375994 +0000 UTC m=+0.397181433,LastTimestamp:2026-08-02 09:32:56.682375994 +0000 UTC m=+0.397181433,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-232.ec2.internal,}"
Aug 02 09:32:56.694946 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.694933    2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Aug 02 09:32:56.695016 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.694950    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Aug 02 09:32:56.695016 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:32:56.695001    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-232.ec2.internal\" not found"
Aug 02 09:32:56.695676 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.695658    2564 factory.go:55] Registering systemd factory
Aug 02 09:32:56.695676 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.695678    2564 factory.go:223] Registration of the systemd container factory successfully
Aug 02 09:32:56.695835 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.695703    2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Aug 02 09:32:56.695835 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.695717    2564 volume_manager.go:311] "The desired_state_of_world populator starts"
Aug 02 09:32:56.695835 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.695730    2564 volume_manager.go:313] "Starting Kubelet Volume Manager"
Aug 02 09:32:56.695835 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.695831    2564 reconstruct.go:97] "Volume reconstruction finished"
Aug 02 09:32:56.696010 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.695840    2564 reconciler.go:29] "Reconciler: start to sync state"
Aug 02 09:32:56.696010 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.695853    2564 factory.go:153] Registering CRI-O factory
Aug 02 09:32:56.696010 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.695865    2564 factory.go:223] Registration of the crio container factory successfully
Aug 02 09:32:56.696010 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.695910    2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Aug 02 09:32:56.696010 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.695933    2564 factory.go:103] Registering Raw factory
Aug 02 09:32:56.696010 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.695950    2564 manager.go:1196] Started watching for new ooms in manager
Aug 02 09:32:56.697273 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.697255    2564 manager.go:349] Starting recovery of all containers
Aug 02 09:32:56.706081 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.706061    2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 02 09:32:56.708809 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:32:56.708789    2564 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-141-232.ec2.internal\" not found" node="ip-10-0-141-232.ec2.internal"
Aug 02 09:32:56.709536 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.709515    2564 manager.go:354] Recovery completed
Aug 02 09:32:56.714150 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.714134    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 02 09:32:56.715845 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.715832    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-232.ec2.internal" event="NodeHasSufficientMemory"
Aug 02 09:32:56.715911 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.715858    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-232.ec2.internal" event="NodeHasNoDiskPressure"
Aug 02 09:32:56.715911 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.715867    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-232.ec2.internal" event="NodeHasSufficientPID"
Aug 02 09:32:56.716386 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.716373    2564 cpu_manager.go:222] "Starting CPU manager" policy="none"
Aug 02 09:32:56.716441 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.716387    2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Aug 02 09:32:56.716441 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.716373    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 02 09:32:56.716441 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.716409    2564 state_mem.go:36] "Initialized new in-memory state store"
Aug 02 09:32:56.717072 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.717060    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-232.ec2.internal" event="NodeHasSufficientMemory"
Aug 02 09:32:56.717121 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.717085    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-232.ec2.internal" event="NodeHasNoDiskPressure"
Aug 02 09:32:56.717121 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.717099    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-232.ec2.internal" event="NodeHasSufficientPID"
Aug 02 09:32:56.718512 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.718501    2564 policy_none.go:49] "None policy: Start"
Aug 02 09:32:56.718554 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.718516    2564 memory_manager.go:187] "Starting memorymanager" policy="None"
Aug 02 09:32:56.718554 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.718526    2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Aug 02 09:32:56.720433 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.720423    2564 policy_none.go:47] "Start"
Aug 02 09:32:56.753789 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.753773    2564 manager.go:337] "Starting Device Plugin manager"
Aug 02 09:32:56.779462 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:32:56.753810    2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Aug 02 09:32:56.779462 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.753823    2564 server.go:87] "Starting device plugin registration server"
Aug 02 09:32:56.779462 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.754053    2564 eviction_manager.go:189] "Eviction manager: starting control loop"
Aug 02 09:32:56.779462 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.754064    2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Aug 02 09:32:56.779462 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.754161    2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Aug 02 09:32:56.779462 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.754259    2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Aug 02 09:32:56.779462 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.754269    2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Aug 02 09:32:56.779462 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:32:56.754673    2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Aug 02 09:32:56.779462 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:32:56.754723    2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-232.ec2.internal\" not found"
Aug 02 09:32:56.779462 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.772656    2564 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2601 into the system container \"/system.slice\": <nil>"
Aug 02 09:32:56.854166 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.854112    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 02 09:32:56.856954 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.856939    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-232.ec2.internal" event="NodeHasSufficientMemory"
Aug 02 09:32:56.857026 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.856968    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-232.ec2.internal" event="NodeHasNoDiskPressure"
Aug 02 09:32:56.857026 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.856983    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-232.ec2.internal" event="NodeHasSufficientPID"
Aug 02 09:32:56.857026 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.857015    2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-232.ec2.internal"
Aug 02 09:32:56.864743 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.864728    2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-232.ec2.internal"
Aug 02 09:32:56.864806 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:32:56.864751    2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-232.ec2.internal\": node \"ip-10-0-141-232.ec2.internal\" not found"
Aug 02 09:32:56.873405 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.873378    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Aug 02 09:32:56.874555 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.874539    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Aug 02 09:32:56.874645 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.874570    2564 status_manager.go:244] "Starting to sync pod status with apiserver"
Aug 02 09:32:56.874645 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.874589    2564 kubelet.go:2448] "Starting kubelet main sync loop"
Aug 02 09:32:56.874645 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:32:56.874620    2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Aug 02 09:32:56.876913 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.876720    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 02 09:32:56.879132 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:32:56.879117    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-232.ec2.internal\" not found"
Aug 02 09:32:56.975625 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.975596    2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-232.ec2.internal"]
Aug 02 09:32:56.975701 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.975664    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 02 09:32:56.977134 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.977115    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-232.ec2.internal" event="NodeHasSufficientMemory"
Aug 02 09:32:56.977209 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.977147    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-232.ec2.internal" event="NodeHasNoDiskPressure"
Aug 02 09:32:56.977209 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.977158    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-232.ec2.internal" event="NodeHasSufficientPID"
Aug 02 09:32:56.979225 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:32:56.979211    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-232.ec2.internal\" not found"
Aug 02 09:32:56.979451 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.979440    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 02 09:32:56.979572 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.979558    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal"
Aug 02 09:32:56.979625 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.979588    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 02 09:32:56.980071 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.980049    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-232.ec2.internal" event="NodeHasSufficientMemory"
Aug 02 09:32:56.980071 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.980061    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-232.ec2.internal" event="NodeHasSufficientMemory"
Aug 02 09:32:56.980158 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.980076    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-232.ec2.internal" event="NodeHasNoDiskPressure"
Aug 02 09:32:56.980158 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.980082    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-232.ec2.internal" event="NodeHasNoDiskPressure"
Aug 02 09:32:56.980158 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.980087    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-232.ec2.internal" event="NodeHasSufficientPID"
Aug 02 09:32:56.980158 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.980092    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-232.ec2.internal" event="NodeHasSufficientPID"
Aug 02 09:32:56.982243 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.982215    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-232.ec2.internal"
Aug 02 09:32:56.982319 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.982250    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 02 09:32:56.982846 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.982833    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-232.ec2.internal" event="NodeHasSufficientMemory"
Aug 02 09:32:56.982910 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.982856    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-232.ec2.internal" event="NodeHasNoDiskPressure"
Aug 02 09:32:56.982910 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.982867    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-232.ec2.internal" event="NodeHasSufficientPID"
Aug 02 09:32:56.996907 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.996891    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5fa2085f44de3c0708af14f57a9f26c5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal\" (UID: \"5fa2085f44de3c0708af14f57a9f26c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal"
Aug 02 09:32:56.996980 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.996913    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5fa2085f44de3c0708af14f57a9f26c5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal\" (UID: \"5fa2085f44de3c0708af14f57a9f26c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal"
Aug 02 09:32:56.996980 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:56.996932    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ce69cf0400f5bcb61dc1156a46d82c8f-config\") pod \"kube-apiserver-proxy-ip-10-0-141-232.ec2.internal\" (UID: \"ce69cf0400f5bcb61dc1156a46d82c8f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-232.ec2.internal"
Aug 02 09:32:57.015764 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:32:57.015744    2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-232.ec2.internal\" not found" node="ip-10-0-141-232.ec2.internal"
Aug 02 09:32:57.020045 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:32:57.020031    2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-232.ec2.internal\" not found" node="ip-10-0-141-232.ec2.internal"
Aug 02 09:32:57.079701 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:32:57.079682    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-232.ec2.internal\" not found"
Aug 02 09:32:57.097302 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:57.097281    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5fa2085f44de3c0708af14f57a9f26c5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal\" (UID: \"5fa2085f44de3c0708af14f57a9f26c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal"
Aug 02 09:32:57.097390 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:57.097307    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5fa2085f44de3c0708af14f57a9f26c5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal\" (UID: \"5fa2085f44de3c0708af14f57a9f26c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal"
Aug 02 09:32:57.097390 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:57.097333    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ce69cf0400f5bcb61dc1156a46d82c8f-config\") pod \"kube-apiserver-proxy-ip-10-0-141-232.ec2.internal\" (UID: \"ce69cf0400f5bcb61dc1156a46d82c8f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-232.ec2.internal"
Aug 02 09:32:57.097488 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:57.097386    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5fa2085f44de3c0708af14f57a9f26c5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal\" (UID: \"5fa2085f44de3c0708af14f57a9f26c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal"
Aug 02 09:32:57.097488 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:57.097461    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ce69cf0400f5bcb61dc1156a46d82c8f-config\") pod \"kube-apiserver-proxy-ip-10-0-141-232.ec2.internal\" (UID: \"ce69cf0400f5bcb61dc1156a46d82c8f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-232.ec2.internal"
Aug 02 09:32:57.097566 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:57.097496    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5fa2085f44de3c0708af14f57a9f26c5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal\" (UID: \"5fa2085f44de3c0708af14f57a9f26c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal"
Aug 02 09:32:57.180691 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:32:57.180631    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-232.ec2.internal\" not found"
Aug 02 09:32:57.281125 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:32:57.281096    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-232.ec2.internal\" not found"
Aug 02 09:32:57.319552 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:57.319534    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal"
Aug 02 09:32:57.323615 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:57.323602    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-232.ec2.internal"
Aug 02 09:32:57.381763 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:32:57.381728    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-232.ec2.internal\" not found"
Aug 02 09:32:57.482254 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:32:57.482182    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-232.ec2.internal\" not found"
Aug 02 09:32:57.582906 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:32:57.582878    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-232.ec2.internal\" not found"
Aug 02 09:32:57.612380 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:57.612361    2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 02 09:32:57.626355 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:57.626333    2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Aug 02 09:32:57.626466 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:57.626449    2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Aug 02 09:32:57.626512 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:57.626496    2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Aug 02 09:32:57.626548 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:57.626496    2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Aug 02 09:32:57.682988 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:32:57.682961    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-232.ec2.internal\" not found"
Aug 02 09:32:57.693546 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:57.693510    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-08-01 09:27:57 +0000 UTC" deadline="2028-03-25 00:01:52.259101274 +0000 UTC"
Aug 02 09:32:57.693546 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:57.693544    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14414h28m54.565560638s"
Aug 02 09:32:57.784180 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:32:57.784126    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-232.ec2.internal\" not found"
Aug 02 09:32:57.884551 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:32:57.884526    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-232.ec2.internal\" not found"
Aug 02 09:32:57.966056 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:57.966028    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce69cf0400f5bcb61dc1156a46d82c8f.slice/crio-68c4767786d806371948acc799561487df426b34f14588d8272eb06352879585 WatchSource:0}: Error finding container 68c4767786d806371948acc799561487df426b34f14588d8272eb06352879585: Status 404 returned error can't find the container with id 68c4767786d806371948acc799561487df426b34f14588d8272eb06352879585
Aug 02 09:32:57.970914 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:57.970899    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 02 09:32:57.974542 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:57.974525    2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 02 09:32:57.976047 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:32:57.976026    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fa2085f44de3c0708af14f57a9f26c5.slice/crio-f93daceacec8d7a3e19de953f14ed2b5ff75b9ff3d5e3f1b0f5afb7cfa35ff56 WatchSource:0}: Error finding container f93daceacec8d7a3e19de953f14ed2b5ff75b9ff3d5e3f1b0f5afb7cfa35ff56: Status 404 returned error can't find the container with id f93daceacec8d7a3e19de953f14ed2b5ff75b9ff3d5e3f1b0f5afb7cfa35ff56
Aug 02 09:32:57.995883 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:57.995865    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal"
Aug 02 09:32:58.006846 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.006825    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Aug 02 09:32:58.008123 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.008109    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-232.ec2.internal"
Aug 02 09:32:58.013482 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.013466    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Aug 02 09:32:58.491483 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.491453    2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 02 09:32:58.674147 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.674113    2564 apiserver.go:52] "Watching apiserver"
Aug 02 09:32:58.683853 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.683488    2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Aug 02 09:32:58.685268 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.684843    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-cs5vh","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfhxd","openshift-cluster-node-tuning-operator/tuned-fj9zj","openshift-dns/node-resolver-4f6pd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal","openshift-multus/multus-additional-cni-plugins-qgz2q","openshift-multus/network-metrics-daemon-v5mrn","openshift-network-diagnostics/network-check-target-m9jmd","kube-system/konnectivity-agent-fhzz9","kube-system/kube-apiserver-proxy-ip-10-0-141-232.ec2.internal","openshift-image-registry/node-ca-8lm5w","openshift-multus/multus-ht722","openshift-network-operator/iptables-alerter-878zb","openshift-ovn-kubernetes/ovnkube-node-59x9w"]
Aug 02 09:32:58.687729 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.687707    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5mrn"
Aug 02 09:32:58.687831 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:32:58.687804    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5mrn" podUID="2068f737-ee2f-4310-977e-27cb7e1d4f73"
Aug 02 09:32:58.690493 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.690014    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8lm5w"
Aug 02 09:32:58.692091 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.691879    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-gnhr4\""
Aug 02 09:32:58.692948 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.692321    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fj9zj"
Aug 02 09:32:58.692948 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.692516    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Aug 02 09:32:58.692948 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.692796    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Aug 02 09:32:58.692948 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.692824    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Aug 02 09:32:58.695608 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.695583    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Aug 02 09:32:58.695770 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.695731    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-j95mm\""
Aug 02 09:32:58.695915 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.695899    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Aug 02 09:32:58.695974 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.695927    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Aug 02 09:32:58.696663 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.696626    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4f6pd"
Aug 02 09:32:58.697230 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.697214    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.699053 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.698714    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Aug 02 09:32:58.699053 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.698812    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Aug 02 09:32:58.699053 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.699039    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-sdzkv\""
Aug 02 09:32:58.699401 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.699384    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Aug 02 09:32:58.699489 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.699476    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-smt4n\""
Aug 02 09:32:58.700071 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.699626    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Aug 02 09:32:58.700071 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.699696    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Aug 02 09:32:58.700071 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.699972    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Aug 02 09:32:58.700071 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.700045    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Aug 02 09:32:58.700409 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.700049    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Aug 02 09:32:58.700543 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.700513    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qgz2q"
Aug 02 09:32:58.702594 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.702574    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Aug 02 09:32:58.702797 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.702782    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Aug 02 09:32:58.702882 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.702824    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fhzz9"
Aug 02 09:32:58.703511 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.703493    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Aug 02 09:32:58.703627 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.703576    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nv7dm\""
Aug 02 09:32:58.703696 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.703634    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Aug 02 09:32:58.704961 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.704912    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Aug 02 09:32:58.705747 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.705083    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Aug 02 09:32:58.705747 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.705557    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-878zb"
Aug 02 09:32:58.706565 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.706401    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Aug 02 09:32:58.707394 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.707068    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qfb6w\""
Aug 02 09:32:58.707702 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.707554    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Aug 02 09:32:58.707847 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.707829    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Aug 02 09:32:58.708064 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.708045    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-n9qjp\""
Aug 02 09:32:58.708262 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.708244    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Aug 02 09:32:58.708472 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.708455    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Aug 02 09:32:58.708739 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.708718    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-run-ovn\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.708812 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.708758    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-node-log\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.708812 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.708769    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfhxd"
Aug 02 09:32:58.708812 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.708789    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/032156ef-c7a4-4cd5-a991-4c91f2a67b4b-tmp-dir\") pod \"node-resolver-4f6pd\" (UID: \"032156ef-c7a4-4cd5-a991-4c91f2a67b4b\") " pod="openshift-dns/node-resolver-4f6pd"
Aug 02 09:32:58.708967 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.708815    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-host-run-ovn-kubernetes\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.708967 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.708842    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-host-cni-netd\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.708967 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.708866    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6b0cc25e-0367-4ac6-b094-1caf4d2023ab-host\") pod \"node-ca-8lm5w\" (UID: \"6b0cc25e-0367-4ac6-b094-1caf4d2023ab\") " pod="openshift-image-registry/node-ca-8lm5w"
Aug 02 09:32:58.708967 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.708891    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/95220006-a44c-46f9-b380-f771acbd1eac-run\") pod \"tuned-fj9zj\" (UID: \"95220006-a44c-46f9-b380-f771acbd1eac\") " pod="openshift-cluster-node-tuning-operator/tuned-fj9zj"
Aug 02 09:32:58.708967 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.708919    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.708967 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.708950    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm7qx\" (UniqueName: \"kubernetes.io/projected/2068f737-ee2f-4310-977e-27cb7e1d4f73-kube-api-access-dm7qx\") pod \"network-metrics-daemon-v5mrn\" (UID: \"2068f737-ee2f-4310-977e-27cb7e1d4f73\") " pod="openshift-multus/network-metrics-daemon-v5mrn"
Aug 02 09:32:58.709313 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.708977    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95220006-a44c-46f9-b380-f771acbd1eac-var-lib-kubelet\") pod \"tuned-fj9zj\" (UID: \"95220006-a44c-46f9-b380-f771acbd1eac\") " pod="openshift-cluster-node-tuning-operator/tuned-fj9zj"
Aug 02 09:32:58.709313 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.709004    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/032156ef-c7a4-4cd5-a991-4c91f2a67b4b-hosts-file\") pod \"node-resolver-4f6pd\" (UID: \"032156ef-c7a4-4cd5-a991-4c91f2a67b4b\") " pod="openshift-dns/node-resolver-4f6pd"
Aug 02 09:32:58.709313 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.709029    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-run-systemd\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.709313 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.709065    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-host-cni-bin\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.709313 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.709094    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-ovnkube-config\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.709313 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.709120    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/95220006-a44c-46f9-b380-f771acbd1eac-etc-sysctl-conf\") pod \"tuned-fj9zj\" (UID: \"95220006-a44c-46f9-b380-f771acbd1eac\") " pod="openshift-cluster-node-tuning-operator/tuned-fj9zj"
Aug 02 09:32:58.709313 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.709136    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-host-kubelet\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.709313 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.709150    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-systemd-units\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.709313 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.709166    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-var-lib-openvswitch\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.709313 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.709180    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-log-socket\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.709313 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.709193    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-env-overrides\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.709313 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.709208    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/95220006-a44c-46f9-b380-f771acbd1eac-etc-sysconfig\") pod \"tuned-fj9zj\" (UID: \"95220006-a44c-46f9-b380-f771acbd1eac\") " pod="openshift-cluster-node-tuning-operator/tuned-fj9zj"
Aug 02 09:32:58.709313 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.709222    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/95220006-a44c-46f9-b380-f771acbd1eac-etc-systemd\") pod \"tuned-fj9zj\" (UID: \"95220006-a44c-46f9-b380-f771acbd1eac\") " pod="openshift-cluster-node-tuning-operator/tuned-fj9zj"
Aug 02 09:32:58.709313 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.709236    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95220006-a44c-46f9-b380-f771acbd1eac-tmp\") pod \"tuned-fj9zj\" (UID: \"95220006-a44c-46f9-b380-f771acbd1eac\") " pod="openshift-cluster-node-tuning-operator/tuned-fj9zj"
Aug 02 09:32:58.709313 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.709255    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c47dk\" (UniqueName: \"kubernetes.io/projected/95220006-a44c-46f9-b380-f771acbd1eac-kube-api-access-c47dk\") pod \"tuned-fj9zj\" (UID: \"95220006-a44c-46f9-b380-f771acbd1eac\") " pod="openshift-cluster-node-tuning-operator/tuned-fj9zj"
Aug 02 09:32:58.709313 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.709293    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-etc-openvswitch\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.710170 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.709318    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssdjm\" (UniqueName: \"kubernetes.io/projected/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-kube-api-access-ssdjm\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.710170 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.709344    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2068f737-ee2f-4310-977e-27cb7e1d4f73-metrics-certs\") pod \"network-metrics-daemon-v5mrn\" (UID: \"2068f737-ee2f-4310-977e-27cb7e1d4f73\") " pod="openshift-multus/network-metrics-daemon-v5mrn"
Aug 02 09:32:58.710170 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.709368    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8f7j\" (UniqueName: \"kubernetes.io/projected/6b0cc25e-0367-4ac6-b094-1caf4d2023ab-kube-api-access-q8f7j\") pod \"node-ca-8lm5w\" (UID: \"6b0cc25e-0367-4ac6-b094-1caf4d2023ab\") " pod="openshift-image-registry/node-ca-8lm5w"
Aug 02 09:32:58.710170 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.709391    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/95220006-a44c-46f9-b380-f771acbd1eac-etc-modprobe-d\") pod \"tuned-fj9zj\" (UID: \"95220006-a44c-46f9-b380-f771acbd1eac\") " pod="openshift-cluster-node-tuning-operator/tuned-fj9zj"
Aug 02 09:32:58.710170 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.709420    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-host-slash\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.710170 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.709446    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-ovn-node-metrics-cert\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.710170 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.709490    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-ovnkube-script-lib\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.710170 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.709520    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6b0cc25e-0367-4ac6-b094-1caf4d2023ab-serviceca\") pod \"node-ca-8lm5w\" (UID: \"6b0cc25e-0367-4ac6-b094-1caf4d2023ab\") " pod="openshift-image-registry/node-ca-8lm5w"
Aug 02 09:32:58.710170 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.709542    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/95220006-a44c-46f9-b380-f771acbd1eac-etc-sysctl-d\") pod \"tuned-fj9zj\" (UID: \"95220006-a44c-46f9-b380-f771acbd1eac\") " pod="openshift-cluster-node-tuning-operator/tuned-fj9zj"
Aug 02 09:32:58.710170 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.709565    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/95220006-a44c-46f9-b380-f771acbd1eac-sys\") pod \"tuned-fj9zj\" (UID: \"95220006-a44c-46f9-b380-f771acbd1eac\") " pod="openshift-cluster-node-tuning-operator/tuned-fj9zj"
Aug 02 09:32:58.710170 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.709602    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/95220006-a44c-46f9-b380-f771acbd1eac-lib-modules\") pod \"tuned-fj9zj\" (UID: \"95220006-a44c-46f9-b380-f771acbd1eac\") " pod="openshift-cluster-node-tuning-operator/tuned-fj9zj"
Aug 02 09:32:58.710170 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.709624    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95220006-a44c-46f9-b380-f771acbd1eac-host\") pod \"tuned-fj9zj\" (UID: \"95220006-a44c-46f9-b380-f771acbd1eac\") " pod="openshift-cluster-node-tuning-operator/tuned-fj9zj"
Aug 02 09:32:58.710170 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.709646    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/95220006-a44c-46f9-b380-f771acbd1eac-etc-tuned\") pod \"tuned-fj9zj\" (UID: \"95220006-a44c-46f9-b380-f771acbd1eac\") " pod="openshift-cluster-node-tuning-operator/tuned-fj9zj"
Aug 02 09:32:58.710170 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.709670    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tltfz\" (UniqueName: \"kubernetes.io/projected/032156ef-c7a4-4cd5-a991-4c91f2a67b4b-kube-api-access-tltfz\") pod \"node-resolver-4f6pd\" (UID: \"032156ef-c7a4-4cd5-a991-4c91f2a67b4b\") " pod="openshift-dns/node-resolver-4f6pd"
Aug 02 09:32:58.710170 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.709693    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-host-run-netns\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.710170 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.709717    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95220006-a44c-46f9-b380-f771acbd1eac-etc-kubernetes\") pod \"tuned-fj9zj\" (UID: \"95220006-a44c-46f9-b380-f771acbd1eac\") " pod="openshift-cluster-node-tuning-operator/tuned-fj9zj"
Aug 02 09:32:58.710170 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.709744    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-run-openvswitch\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.711031 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.710862    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Aug 02 09:32:58.711191 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.711171    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vztqm\""
Aug 02 09:32:58.711585 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.711565    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Aug 02 09:32:58.711883 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.711864    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Aug 02 09:32:58.714277 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.713684    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jmd"
Aug 02 09:32:58.714277 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:32:58.713760    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jmd" podUID="3c47caf2-5d96-4343-ab9a-5da7195e209c"
Aug 02 09:32:58.717901 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.717882    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cs5vh"
Aug 02 09:32:58.717995 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:32:58.717951    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cs5vh" podUID="e6944ae2-aaed-4a4a-9c46-dda99772fa54"
Aug 02 09:32:58.720810 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.720525    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ht722"
Aug 02 09:32:58.723289 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.722791    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-f6b46\""
Aug 02 09:32:58.723289 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.723127    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Aug 02 09:32:58.724374 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.724353    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vvzk5"
Aug 02 09:32:58.734043 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.734023    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vvzk5"
Aug 02 09:32:58.797308 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.797289    2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Aug 02 09:32:58.810399 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.810374    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-host-run-netns\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.810521 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.810417    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0fb6399d-1991-4b8e-920c-a1086603b643-cnibin\") pod \"multus-additional-cni-plugins-qgz2q\" (UID: \"0fb6399d-1991-4b8e-920c-a1086603b643\") " pod="openshift-multus/multus-additional-cni-plugins-qgz2q"
Aug 02 09:32:58.810521 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.810443    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9901b738-6f94-48a9-9282-a051ce9d37ae-multus-socket-dir-parent\") pod \"multus-ht722\" (UID: \"9901b738-6f94-48a9-9282-a051ce9d37ae\") " pod="openshift-multus/multus-ht722"
Aug 02 09:32:58.810521 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.810469    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9901b738-6f94-48a9-9282-a051ce9d37ae-host-run-k8s-cni-cncf-io\") pod \"multus-ht722\" (UID: \"9901b738-6f94-48a9-9282-a051ce9d37ae\") " pod="openshift-multus/multus-ht722"
Aug 02 09:32:58.810521 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.810498    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-node-log\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.810521 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.810498    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-host-run-netns\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.810793 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.810522    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9901b738-6f94-48a9-9282-a051ce9d37ae-host-var-lib-cni-bin\") pod \"multus-ht722\" (UID: \"9901b738-6f94-48a9-9282-a051ce9d37ae\") " pod="openshift-multus/multus-ht722"
Aug 02 09:32:58.810793 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.810552    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9901b738-6f94-48a9-9282-a051ce9d37ae-host-run-multus-certs\") pod \"multus-ht722\" (UID: \"9901b738-6f94-48a9-9282-a051ce9d37ae\") " pod="openshift-multus/multus-ht722"
Aug 02 09:32:58.810793 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.810578    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0fb6399d-1991-4b8e-920c-a1086603b643-cni-binary-copy\") pod \"multus-additional-cni-plugins-qgz2q\" (UID: \"0fb6399d-1991-4b8e-920c-a1086603b643\") " pod="openshift-multus/multus-additional-cni-plugins-qgz2q"
Aug 02 09:32:58.810793 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.810607    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-node-log\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.810793 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.810621    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6ddc3ad8-170f-49b5-8a1c-6fe966dd658a-etc-selinux\") pod \"aws-ebs-csi-driver-node-tfhxd\" (UID: \"6ddc3ad8-170f-49b5-8a1c-6fe966dd658a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfhxd"
Aug 02 09:32:58.810793 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.810661    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9901b738-6f94-48a9-9282-a051ce9d37ae-host-var-lib-cni-multus\") pod \"multus-ht722\" (UID: \"9901b738-6f94-48a9-9282-a051ce9d37ae\") " pod="openshift-multus/multus-ht722"
Aug 02 09:32:58.810793 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.810694    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.810793 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.810734    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-842wd\" (UniqueName: \"kubernetes.io/projected/6ddc3ad8-170f-49b5-8a1c-6fe966dd658a-kube-api-access-842wd\") pod \"aws-ebs-csi-driver-node-tfhxd\" (UID: \"6ddc3ad8-170f-49b5-8a1c-6fe966dd658a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfhxd"
Aug 02 09:32:58.810793 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.810786    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.811178 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.810838    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e6944ae2-aaed-4a4a-9c46-dda99772fa54-original-pull-secret\") pod \"global-pull-secret-syncer-cs5vh\" (UID: \"e6944ae2-aaed-4a4a-9c46-dda99772fa54\") " pod="kube-system/global-pull-secret-syncer-cs5vh"
Aug 02 09:32:58.811178 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.810865    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9901b738-6f94-48a9-9282-a051ce9d37ae-host-var-lib-kubelet\") pod \"multus-ht722\" (UID: \"9901b738-6f94-48a9-9282-a051ce9d37ae\") " pod="openshift-multus/multus-ht722"
Aug 02 09:32:58.811178 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.810889    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9901b738-6f94-48a9-9282-a051ce9d37ae-hostroot\") pod \"multus-ht722\" (UID: \"9901b738-6f94-48a9-9282-a051ce9d37ae\") " pod="openshift-multus/multus-ht722"
Aug 02 09:32:58.811178 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.810913    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9901b738-6f94-48a9-9282-a051ce9d37ae-multus-conf-dir\") pod \"multus-ht722\" (UID: \"9901b738-6f94-48a9-9282-a051ce9d37ae\") " pod="openshift-multus/multus-ht722"
Aug 02 09:32:58.811178 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.810937    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3f97d5c9-1a25-43c5-8050-9c619be890e4-agent-certs\") pod \"konnectivity-agent-fhzz9\" (UID: \"3f97d5c9-1a25-43c5-8050-9c619be890e4\") " pod="kube-system/konnectivity-agent-fhzz9"
Aug 02 09:32:58.811178 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.810964    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95220006-a44c-46f9-b380-f771acbd1eac-var-lib-kubelet\") pod \"tuned-fj9zj\" (UID: \"95220006-a44c-46f9-b380-f771acbd1eac\") " pod="openshift-cluster-node-tuning-operator/tuned-fj9zj"
Aug 02 09:32:58.811178 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.810988    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-run-systemd\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.811178 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.811013    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-ovnkube-config\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.811178 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.811055    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95220006-a44c-46f9-b380-f771acbd1eac-var-lib-kubelet\") pod \"tuned-fj9zj\" (UID: \"95220006-a44c-46f9-b380-f771acbd1eac\") " pod="openshift-cluster-node-tuning-operator/tuned-fj9zj"
Aug 02 09:32:58.811178 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.811128    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-run-systemd\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.811178 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.811173    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6ddc3ad8-170f-49b5-8a1c-6fe966dd658a-socket-dir\") pod \"aws-ebs-csi-driver-node-tfhxd\" (UID: \"6ddc3ad8-170f-49b5-8a1c-6fe966dd658a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfhxd"
Aug 02 09:32:58.811690 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.811209    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9901b738-6f94-48a9-9282-a051ce9d37ae-host-run-netns\") pod \"multus-ht722\" (UID: \"9901b738-6f94-48a9-9282-a051ce9d37ae\") " pod="openshift-multus/multus-ht722"
Aug 02 09:32:58.811690 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.811261    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/95220006-a44c-46f9-b380-f771acbd1eac-etc-sysctl-conf\") pod \"tuned-fj9zj\" (UID: \"95220006-a44c-46f9-b380-f771acbd1eac\") " pod="openshift-cluster-node-tuning-operator/tuned-fj9zj"
Aug 02 09:32:58.811690 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.811301    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-host-kubelet\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.811690 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.811325    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-var-lib-openvswitch\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.811690 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.811349    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6ddc3ad8-170f-49b5-8a1c-6fe966dd658a-sys-fs\") pod \"aws-ebs-csi-driver-node-tfhxd\" (UID: \"6ddc3ad8-170f-49b5-8a1c-6fe966dd658a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfhxd"
Aug 02 09:32:58.811690 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.811374    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e6944ae2-aaed-4a4a-9c46-dda99772fa54-kubelet-config\") pod \"global-pull-secret-syncer-cs5vh\" (UID: \"e6944ae2-aaed-4a4a-9c46-dda99772fa54\") " pod="kube-system/global-pull-secret-syncer-cs5vh"
Aug 02 09:32:58.811690 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.811425    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/95220006-a44c-46f9-b380-f771acbd1eac-etc-sysconfig\") pod \"tuned-fj9zj\" (UID: \"95220006-a44c-46f9-b380-f771acbd1eac\") " pod="openshift-cluster-node-tuning-operator/tuned-fj9zj"
Aug 02 09:32:58.811690 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.811451    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/95220006-a44c-46f9-b380-f771acbd1eac-etc-systemd\") pod \"tuned-fj9zj\" (UID: \"95220006-a44c-46f9-b380-f771acbd1eac\") " pod="openshift-cluster-node-tuning-operator/tuned-fj9zj"
Aug 02 09:32:58.811690 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.811475    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95220006-a44c-46f9-b380-f771acbd1eac-tmp\") pod \"tuned-fj9zj\" (UID: \"95220006-a44c-46f9-b380-f771acbd1eac\") " pod="openshift-cluster-node-tuning-operator/tuned-fj9zj"
Aug 02 09:32:58.811690 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.811498    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c47dk\" (UniqueName: \"kubernetes.io/projected/95220006-a44c-46f9-b380-f771acbd1eac-kube-api-access-c47dk\") pod \"tuned-fj9zj\" (UID: \"95220006-a44c-46f9-b380-f771acbd1eac\") " pod="openshift-cluster-node-tuning-operator/tuned-fj9zj"
Aug 02 09:32:58.811690 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.811537    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-etc-openvswitch\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.811690 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.811561    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssdjm\" (UniqueName: \"kubernetes.io/projected/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-kube-api-access-ssdjm\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.811690 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.811589    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0fb6399d-1991-4b8e-920c-a1086603b643-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qgz2q\" (UID: \"0fb6399d-1991-4b8e-920c-a1086603b643\") " pod="openshift-multus/multus-additional-cni-plugins-qgz2q"
Aug 02 09:32:58.811690 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.811616    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mb2c\" (UniqueName: \"kubernetes.io/projected/0fb6399d-1991-4b8e-920c-a1086603b643-kube-api-access-4mb2c\") pod \"multus-additional-cni-plugins-qgz2q\" (UID: \"0fb6399d-1991-4b8e-920c-a1086603b643\") " pod="openshift-multus/multus-additional-cni-plugins-qgz2q"
Aug 02 09:32:58.811690 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.811690    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-ovnkube-config\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.812431 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.811733    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2068f737-ee2f-4310-977e-27cb7e1d4f73-metrics-certs\") pod \"network-metrics-daemon-v5mrn\" (UID: \"2068f737-ee2f-4310-977e-27cb7e1d4f73\") " pod="openshift-multus/network-metrics-daemon-v5mrn"
Aug 02 09:32:58.812431 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.811763    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8f7j\" (UniqueName: \"kubernetes.io/projected/6b0cc25e-0367-4ac6-b094-1caf4d2023ab-kube-api-access-q8f7j\") pod \"node-ca-8lm5w\" (UID: \"6b0cc25e-0367-4ac6-b094-1caf4d2023ab\") " pod="openshift-image-registry/node-ca-8lm5w"
Aug 02 09:32:58.812431 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.811785    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-host-kubelet\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.812431 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.811789    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-host-slash\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.812431 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.811829    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-host-slash\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.812431 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.811829    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-ovn-node-metrics-cert\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.812431 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.811877    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2e5695b0-d7f7-4770-93a8-b9bcc79efad1-iptables-alerter-script\") pod \"iptables-alerter-878zb\" (UID: \"2e5695b0-d7f7-4770-93a8-b9bcc79efad1\") " pod="openshift-network-operator/iptables-alerter-878zb"
Aug 02 09:32:58.812431 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.811902    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e6944ae2-aaed-4a4a-9c46-dda99772fa54-dbus\") pod \"global-pull-secret-syncer-cs5vh\" (UID: \"e6944ae2-aaed-4a4a-9c46-dda99772fa54\") " pod="kube-system/global-pull-secret-syncer-cs5vh"
Aug 02 09:32:58.812431 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.811929    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9901b738-6f94-48a9-9282-a051ce9d37ae-cnibin\") pod \"multus-ht722\" (UID: \"9901b738-6f94-48a9-9282-a051ce9d37ae\") " pod="openshift-multus/multus-ht722"
Aug 02 09:32:58.812431 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.812068    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/95220006-a44c-46f9-b380-f771acbd1eac-etc-sysctl-conf\") pod \"tuned-fj9zj\" (UID: \"95220006-a44c-46f9-b380-f771acbd1eac\") " pod="openshift-cluster-node-tuning-operator/tuned-fj9zj"
Aug 02 09:32:58.812431 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.812034    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6b0cc25e-0367-4ac6-b094-1caf4d2023ab-serviceca\") pod \"node-ca-8lm5w\" (UID: \"6b0cc25e-0367-4ac6-b094-1caf4d2023ab\") " pod="openshift-image-registry/node-ca-8lm5w"
Aug 02 09:32:58.812431 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.812132    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/95220006-a44c-46f9-b380-f771acbd1eac-etc-sysctl-d\") pod \"tuned-fj9zj\" (UID: \"95220006-a44c-46f9-b380-f771acbd1eac\") " pod="openshift-cluster-node-tuning-operator/tuned-fj9zj"
Aug 02 09:32:58.812431 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.812145    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/95220006-a44c-46f9-b380-f771acbd1eac-etc-systemd\") pod \"tuned-fj9zj\" (UID: \"95220006-a44c-46f9-b380-f771acbd1eac\") " pod="openshift-cluster-node-tuning-operator/tuned-fj9zj"
Aug 02 09:32:58.812431 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.812161    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/95220006-a44c-46f9-b380-f771acbd1eac-sys\") pod \"tuned-fj9zj\" (UID: \"95220006-a44c-46f9-b380-f771acbd1eac\") " pod="openshift-cluster-node-tuning-operator/tuned-fj9zj"
Aug 02 09:32:58.812431 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.812184    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/95220006-a44c-46f9-b380-f771acbd1eac-lib-modules\") pod \"tuned-fj9zj\" (UID: \"95220006-a44c-46f9-b380-f771acbd1eac\") " pod="openshift-cluster-node-tuning-operator/tuned-fj9zj"
Aug 02 09:32:58.812431 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.812188    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/95220006-a44c-46f9-b380-f771acbd1eac-etc-sysconfig\") pod \"tuned-fj9zj\" (UID: \"95220006-a44c-46f9-b380-f771acbd1eac\") " pod="openshift-cluster-node-tuning-operator/tuned-fj9zj"
Aug 02 09:32:58.812431 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.812199    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95220006-a44c-46f9-b380-f771acbd1eac-host\") pod \"tuned-fj9zj\" (UID: \"95220006-a44c-46f9-b380-f771acbd1eac\") " pod="openshift-cluster-node-tuning-operator/tuned-fj9zj"
Aug 02 09:32:58.812431 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.812218    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/95220006-a44c-46f9-b380-f771acbd1eac-etc-tuned\") pod \"tuned-fj9zj\" (UID: \"95220006-a44c-46f9-b380-f771acbd1eac\") " pod="openshift-cluster-node-tuning-operator/tuned-fj9zj"
Aug 02 09:32:58.813380 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.812242    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tltfz\" (UniqueName: \"kubernetes.io/projected/032156ef-c7a4-4cd5-a991-4c91f2a67b4b-kube-api-access-tltfz\") pod \"node-resolver-4f6pd\" (UID: \"032156ef-c7a4-4cd5-a991-4c91f2a67b4b\") " pod="openshift-dns/node-resolver-4f6pd"
Aug 02 09:32:58.813380 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.812290    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-run-openvswitch\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.813380 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:32:58.812356    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 02 09:32:58.813380 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.812411    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/95220006-a44c-46f9-b380-f771acbd1eac-lib-modules\") pod \"tuned-fj9zj\" (UID: \"95220006-a44c-46f9-b380-f771acbd1eac\") " pod="openshift-cluster-node-tuning-operator/tuned-fj9zj"
Aug 02 09:32:58.813380 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.812422    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/95220006-a44c-46f9-b380-f771acbd1eac-etc-sysctl-d\") pod \"tuned-fj9zj\" (UID: \"95220006-a44c-46f9-b380-f771acbd1eac\") " pod="openshift-cluster-node-tuning-operator/tuned-fj9zj"
Aug 02 09:32:58.813380 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.812466    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/95220006-a44c-46f9-b380-f771acbd1eac-sys\") pod \"tuned-fj9zj\" (UID: \"95220006-a44c-46f9-b380-f771acbd1eac\") " pod="openshift-cluster-node-tuning-operator/tuned-fj9zj"
Aug 02 09:32:58.813380 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.812471    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95220006-a44c-46f9-b380-f771acbd1eac-etc-kubernetes\") pod \"tuned-fj9zj\" (UID: \"95220006-a44c-46f9-b380-f771acbd1eac\") " pod="openshift-cluster-node-tuning-operator/tuned-fj9zj"
Aug 02 09:32:58.813380 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.812503    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-run-ovn\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.813380 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.812535    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-var-lib-openvswitch\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.813380 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.812537    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl4s2\" (UniqueName: \"kubernetes.io/projected/2e5695b0-d7f7-4770-93a8-b9bcc79efad1-kube-api-access-wl4s2\") pod \"iptables-alerter-878zb\" (UID: \"2e5695b0-d7f7-4770-93a8-b9bcc79efad1\") " pod="openshift-network-operator/iptables-alerter-878zb"
Aug 02 09:32:58.813380 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.812472    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-run-openvswitch\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.813380 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:32:58.812588    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2068f737-ee2f-4310-977e-27cb7e1d4f73-metrics-certs podName:2068f737-ee2f-4310-977e-27cb7e1d4f73 nodeName:}" failed. No retries permitted until 2026-08-02 09:32:59.312537264 +0000 UTC m=+3.027342709 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2068f737-ee2f-4310-977e-27cb7e1d4f73-metrics-certs") pod "network-metrics-daemon-v5mrn" (UID: "2068f737-ee2f-4310-977e-27cb7e1d4f73") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 02 09:32:58.813380 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.812619    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9901b738-6f94-48a9-9282-a051ce9d37ae-os-release\") pod \"multus-ht722\" (UID: \"9901b738-6f94-48a9-9282-a051ce9d37ae\") " pod="openshift-multus/multus-ht722"
Aug 02 09:32:58.813380 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.812610    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-run-ovn\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.813380 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.812682    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/032156ef-c7a4-4cd5-a991-4c91f2a67b4b-tmp-dir\") pod \"node-resolver-4f6pd\" (UID: \"032156ef-c7a4-4cd5-a991-4c91f2a67b4b\") " pod="openshift-dns/node-resolver-4f6pd"
Aug 02 09:32:58.813380 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.812710    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-host-run-ovn-kubernetes\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.813380 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.812735    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-host-cni-netd\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.814237 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.812766    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6ddc3ad8-170f-49b5-8a1c-6fe966dd658a-registration-dir\") pod \"aws-ebs-csi-driver-node-tfhxd\" (UID: \"6ddc3ad8-170f-49b5-8a1c-6fe966dd658a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfhxd"
Aug 02 09:32:58.814237 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.812796    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95220006-a44c-46f9-b380-f771acbd1eac-etc-kubernetes\") pod \"tuned-fj9zj\" (UID: \"95220006-a44c-46f9-b380-f771acbd1eac\") " pod="openshift-cluster-node-tuning-operator/tuned-fj9zj"
Aug 02 09:32:58.814237 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.812844    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6b0cc25e-0367-4ac6-b094-1caf4d2023ab-serviceca\") pod \"node-ca-8lm5w\" (UID: \"6b0cc25e-0367-4ac6-b094-1caf4d2023ab\") " pod="openshift-image-registry/node-ca-8lm5w"
Aug 02 09:32:58.814237 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.812862    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9901b738-6f94-48a9-9282-a051ce9d37ae-system-cni-dir\") pod \"multus-ht722\" (UID: \"9901b738-6f94-48a9-9282-a051ce9d37ae\") " pod="openshift-multus/multus-ht722"
Aug 02 09:32:58.814237 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.812133    2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Aug 02 09:32:58.814237 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.812895    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3f97d5c9-1a25-43c5-8050-9c619be890e4-konnectivity-ca\") pod \"konnectivity-agent-fhzz9\" (UID: \"3f97d5c9-1a25-43c5-8050-9c619be890e4\") " pod="kube-system/konnectivity-agent-fhzz9"
Aug 02 09:32:58.814237 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.812945    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtx9x\" (UniqueName: \"kubernetes.io/projected/3c47caf2-5d96-4343-ab9a-5da7195e209c-kube-api-access-mtx9x\") pod \"network-check-target-m9jmd\" (UID: \"3c47caf2-5d96-4343-ab9a-5da7195e209c\") " pod="openshift-network-diagnostics/network-check-target-m9jmd"
Aug 02 09:32:58.814237 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.812931    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-etc-openvswitch\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.814237 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.812997    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6b0cc25e-0367-4ac6-b094-1caf4d2023ab-host\") pod \"node-ca-8lm5w\" (UID: \"6b0cc25e-0367-4ac6-b094-1caf4d2023ab\") " pod="openshift-image-registry/node-ca-8lm5w"
Aug 02 09:32:58.814237 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.812987    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95220006-a44c-46f9-b380-f771acbd1eac-host\") pod \"tuned-fj9zj\" (UID: \"95220006-a44c-46f9-b380-f771acbd1eac\") " pod="openshift-cluster-node-tuning-operator/tuned-fj9zj"
Aug 02 09:32:58.814237 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.813065    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/032156ef-c7a4-4cd5-a991-4c91f2a67b4b-tmp-dir\") pod \"node-resolver-4f6pd\" (UID: \"032156ef-c7a4-4cd5-a991-4c91f2a67b4b\") " pod="openshift-dns/node-resolver-4f6pd"
Aug 02 09:32:58.814237 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.813117    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/95220006-a44c-46f9-b380-f771acbd1eac-run\") pod \"tuned-fj9zj\" (UID: \"95220006-a44c-46f9-b380-f771acbd1eac\") " pod="openshift-cluster-node-tuning-operator/tuned-fj9zj"
Aug 02 09:32:58.814237 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.813130    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6b0cc25e-0367-4ac6-b094-1caf4d2023ab-host\") pod \"node-ca-8lm5w\" (UID: \"6b0cc25e-0367-4ac6-b094-1caf4d2023ab\") " pod="openshift-image-registry/node-ca-8lm5w"
Aug 02 09:32:58.814237 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.813273    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0fb6399d-1991-4b8e-920c-a1086603b643-os-release\") pod \"multus-additional-cni-plugins-qgz2q\" (UID: \"0fb6399d-1991-4b8e-920c-a1086603b643\") " pod="openshift-multus/multus-additional-cni-plugins-qgz2q"
Aug 02 09:32:58.814237 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.813327    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/95220006-a44c-46f9-b380-f771acbd1eac-run\") pod \"tuned-fj9zj\" (UID: \"95220006-a44c-46f9-b380-f771acbd1eac\") " pod="openshift-cluster-node-tuning-operator/tuned-fj9zj"
Aug 02 09:32:58.814237 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.813342    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-host-cni-netd\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.814237 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.813383    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2e5695b0-d7f7-4770-93a8-b9bcc79efad1-host-slash\") pod \"iptables-alerter-878zb\" (UID: \"2e5695b0-d7f7-4770-93a8-b9bcc79efad1\") " pod="openshift-network-operator/iptables-alerter-878zb"
Aug 02 09:32:58.814237 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.813389    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-host-run-ovn-kubernetes\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.815205 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.813418    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9901b738-6f94-48a9-9282-a051ce9d37ae-multus-cni-dir\") pod \"multus-ht722\" (UID: \"9901b738-6f94-48a9-9282-a051ce9d37ae\") " pod="openshift-multus/multus-ht722"
Aug 02 09:32:58.815205 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.813455    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9901b738-6f94-48a9-9282-a051ce9d37ae-multus-daemon-config\") pod \"multus-ht722\" (UID: \"9901b738-6f94-48a9-9282-a051ce9d37ae\") " pod="openshift-multus/multus-ht722"
Aug 02 09:32:58.815205 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.813505    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9901b738-6f94-48a9-9282-a051ce9d37ae-etc-kubernetes\") pod \"multus-ht722\" (UID: \"9901b738-6f94-48a9-9282-a051ce9d37ae\") " pod="openshift-multus/multus-ht722"
Aug 02 09:32:58.815205 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.813529    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mjgs\" (UniqueName: \"kubernetes.io/projected/9901b738-6f94-48a9-9282-a051ce9d37ae-kube-api-access-5mjgs\") pod \"multus-ht722\" (UID: \"9901b738-6f94-48a9-9282-a051ce9d37ae\") " pod="openshift-multus/multus-ht722"
Aug 02 09:32:58.815205 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.813597    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm7qx\" (UniqueName: \"kubernetes.io/projected/2068f737-ee2f-4310-977e-27cb7e1d4f73-kube-api-access-dm7qx\") pod \"network-metrics-daemon-v5mrn\" (UID: \"2068f737-ee2f-4310-977e-27cb7e1d4f73\") " pod="openshift-multus/network-metrics-daemon-v5mrn"
Aug 02 09:32:58.815205 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.813623    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/032156ef-c7a4-4cd5-a991-4c91f2a67b4b-hosts-file\") pod \"node-resolver-4f6pd\" (UID: \"032156ef-c7a4-4cd5-a991-4c91f2a67b4b\") " pod="openshift-dns/node-resolver-4f6pd"
Aug 02 09:32:58.815205 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.813712    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-host-cni-bin\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.815205 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.813759    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-systemd-units\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.815205 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.813797    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-log-socket\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.815205 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.813824    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-env-overrides\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.815205 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.813851    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0fb6399d-1991-4b8e-920c-a1086603b643-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qgz2q\" (UID: \"0fb6399d-1991-4b8e-920c-a1086603b643\") " pod="openshift-multus/multus-additional-cni-plugins-qgz2q"
Aug 02 09:32:58.815205 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.813873    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/032156ef-c7a4-4cd5-a991-4c91f2a67b4b-hosts-file\") pod \"node-resolver-4f6pd\" (UID: \"032156ef-c7a4-4cd5-a991-4c91f2a67b4b\") " pod="openshift-dns/node-resolver-4f6pd"
Aug 02 09:32:58.815205 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.813895    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6ddc3ad8-170f-49b5-8a1c-6fe966dd658a-device-dir\") pod \"aws-ebs-csi-driver-node-tfhxd\" (UID: \"6ddc3ad8-170f-49b5-8a1c-6fe966dd658a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfhxd"
Aug 02 09:32:58.815205 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.813916    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-host-cni-bin\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.815205 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.813921    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9901b738-6f94-48a9-9282-a051ce9d37ae-cni-binary-copy\") pod \"multus-ht722\" (UID: \"9901b738-6f94-48a9-9282-a051ce9d37ae\") " pod="openshift-multus/multus-ht722"
Aug 02 09:32:58.815205 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.813955    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0fb6399d-1991-4b8e-920c-a1086603b643-system-cni-dir\") pod \"multus-additional-cni-plugins-qgz2q\" (UID: \"0fb6399d-1991-4b8e-920c-a1086603b643\") " pod="openshift-multus/multus-additional-cni-plugins-qgz2q"
Aug 02 09:32:58.815205 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.814022    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-log-socket\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.816067 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.814015    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0fb6399d-1991-4b8e-920c-a1086603b643-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qgz2q\" (UID: \"0fb6399d-1991-4b8e-920c-a1086603b643\") " pod="openshift-multus/multus-additional-cni-plugins-qgz2q"
Aug 02 09:32:58.816067 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.814065    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6ddc3ad8-170f-49b5-8a1c-6fe966dd658a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tfhxd\" (UID: \"6ddc3ad8-170f-49b5-8a1c-6fe966dd658a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfhxd"
Aug 02 09:32:58.816067 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.814078    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-systemd-units\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.816067 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.814107    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/95220006-a44c-46f9-b380-f771acbd1eac-etc-modprobe-d\") pod \"tuned-fj9zj\" (UID: \"95220006-a44c-46f9-b380-f771acbd1eac\") " pod="openshift-cluster-node-tuning-operator/tuned-fj9zj"
Aug 02 09:32:58.816067 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.814136    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-ovnkube-script-lib\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.816067 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.814235    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/95220006-a44c-46f9-b380-f771acbd1eac-etc-modprobe-d\") pod \"tuned-fj9zj\" (UID: \"95220006-a44c-46f9-b380-f771acbd1eac\") " pod="openshift-cluster-node-tuning-operator/tuned-fj9zj"
Aug 02 09:32:58.816067 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.814327    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-env-overrides\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.816067 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.814982    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-ovnkube-script-lib\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.817408 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.817386    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95220006-a44c-46f9-b380-f771acbd1eac-tmp\") pod \"tuned-fj9zj\" (UID: \"95220006-a44c-46f9-b380-f771acbd1eac\") " pod="openshift-cluster-node-tuning-operator/tuned-fj9zj"
Aug 02 09:32:58.817736 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.817417    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/95220006-a44c-46f9-b380-f771acbd1eac-etc-tuned\") pod \"tuned-fj9zj\" (UID: \"95220006-a44c-46f9-b380-f771acbd1eac\") " pod="openshift-cluster-node-tuning-operator/tuned-fj9zj"
Aug 02 09:32:58.817873 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.817751    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-ovn-node-metrics-cert\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.825114 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.825025    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8f7j\" (UniqueName: \"kubernetes.io/projected/6b0cc25e-0367-4ac6-b094-1caf4d2023ab-kube-api-access-q8f7j\") pod \"node-ca-8lm5w\" (UID: \"6b0cc25e-0367-4ac6-b094-1caf4d2023ab\") " pod="openshift-image-registry/node-ca-8lm5w"
Aug 02 09:32:58.826439 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.826416    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c47dk\" (UniqueName: \"kubernetes.io/projected/95220006-a44c-46f9-b380-f771acbd1eac-kube-api-access-c47dk\") pod \"tuned-fj9zj\" (UID: \"95220006-a44c-46f9-b380-f771acbd1eac\") " pod="openshift-cluster-node-tuning-operator/tuned-fj9zj"
Aug 02 09:32:58.826439 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.826432    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm7qx\" (UniqueName: \"kubernetes.io/projected/2068f737-ee2f-4310-977e-27cb7e1d4f73-kube-api-access-dm7qx\") pod \"network-metrics-daemon-v5mrn\" (UID: \"2068f737-ee2f-4310-977e-27cb7e1d4f73\") " pod="openshift-multus/network-metrics-daemon-v5mrn"
Aug 02 09:32:58.826830 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.826812    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssdjm\" (UniqueName: \"kubernetes.io/projected/90afa8c6-44be-4641-aa6b-90b51f5c1c6b-kube-api-access-ssdjm\") pod \"ovnkube-node-59x9w\" (UID: \"90afa8c6-44be-4641-aa6b-90b51f5c1c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:58.827744 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.827682    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tltfz\" (UniqueName: \"kubernetes.io/projected/032156ef-c7a4-4cd5-a991-4c91f2a67b4b-kube-api-access-tltfz\") pod \"node-resolver-4f6pd\" (UID: \"032156ef-c7a4-4cd5-a991-4c91f2a67b4b\") " pod="openshift-dns/node-resolver-4f6pd"
Aug 02 09:32:58.879408 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.879364    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal" event={"ID":"5fa2085f44de3c0708af14f57a9f26c5","Type":"ContainerStarted","Data":"f93daceacec8d7a3e19de953f14ed2b5ff75b9ff3d5e3f1b0f5afb7cfa35ff56"}
Aug 02 09:32:58.880522 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.880500    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-232.ec2.internal" event={"ID":"ce69cf0400f5bcb61dc1156a46d82c8f","Type":"ContainerStarted","Data":"68c4767786d806371948acc799561487df426b34f14588d8272eb06352879585"}
Aug 02 09:32:58.915066 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.915039    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0fb6399d-1991-4b8e-920c-a1086603b643-cnibin\") pod \"multus-additional-cni-plugins-qgz2q\" (UID: \"0fb6399d-1991-4b8e-920c-a1086603b643\") " pod="openshift-multus/multus-additional-cni-plugins-qgz2q"
Aug 02 09:32:58.915189 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.915076    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9901b738-6f94-48a9-9282-a051ce9d37ae-multus-socket-dir-parent\") pod \"multus-ht722\" (UID: \"9901b738-6f94-48a9-9282-a051ce9d37ae\") " pod="openshift-multus/multus-ht722"
Aug 02 09:32:58.915189 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.915106    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9901b738-6f94-48a9-9282-a051ce9d37ae-host-run-k8s-cni-cncf-io\") pod \"multus-ht722\" (UID: \"9901b738-6f94-48a9-9282-a051ce9d37ae\") " pod="openshift-multus/multus-ht722"
Aug 02 09:32:58.915189 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.915132    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9901b738-6f94-48a9-9282-a051ce9d37ae-host-var-lib-cni-bin\") pod \"multus-ht722\" (UID: \"9901b738-6f94-48a9-9282-a051ce9d37ae\") " pod="openshift-multus/multus-ht722"
Aug 02 09:32:58.915189 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.915151    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9901b738-6f94-48a9-9282-a051ce9d37ae-host-run-multus-certs\") pod \"multus-ht722\" (UID: \"9901b738-6f94-48a9-9282-a051ce9d37ae\") " pod="openshift-multus/multus-ht722"
Aug 02 09:32:58.915189 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.915167    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0fb6399d-1991-4b8e-920c-a1086603b643-cni-binary-copy\") pod \"multus-additional-cni-plugins-qgz2q\" (UID: \"0fb6399d-1991-4b8e-920c-a1086603b643\") " pod="openshift-multus/multus-additional-cni-plugins-qgz2q"
Aug 02 09:32:58.915189 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.915182    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6ddc3ad8-170f-49b5-8a1c-6fe966dd658a-etc-selinux\") pod \"aws-ebs-csi-driver-node-tfhxd\" (UID: \"6ddc3ad8-170f-49b5-8a1c-6fe966dd658a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfhxd"
Aug 02 09:32:58.917128 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.915202    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9901b738-6f94-48a9-9282-a051ce9d37ae-host-var-lib-cni-multus\") pod \"multus-ht722\" (UID: \"9901b738-6f94-48a9-9282-a051ce9d37ae\") " pod="openshift-multus/multus-ht722"
Aug 02 09:32:58.917128 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.915230    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-842wd\" (UniqueName: \"kubernetes.io/projected/6ddc3ad8-170f-49b5-8a1c-6fe966dd658a-kube-api-access-842wd\") pod \"aws-ebs-csi-driver-node-tfhxd\" (UID: \"6ddc3ad8-170f-49b5-8a1c-6fe966dd658a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfhxd"
Aug 02 09:32:58.917128 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.915257    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e6944ae2-aaed-4a4a-9c46-dda99772fa54-original-pull-secret\") pod \"global-pull-secret-syncer-cs5vh\" (UID: \"e6944ae2-aaed-4a4a-9c46-dda99772fa54\") " pod="kube-system/global-pull-secret-syncer-cs5vh"
Aug 02 09:32:58.917128 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.915297    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9901b738-6f94-48a9-9282-a051ce9d37ae-host-var-lib-kubelet\") pod \"multus-ht722\" (UID: \"9901b738-6f94-48a9-9282-a051ce9d37ae\") " pod="openshift-multus/multus-ht722"
Aug 02 09:32:58.917128 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.915320    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9901b738-6f94-48a9-9282-a051ce9d37ae-hostroot\") pod \"multus-ht722\" (UID: \"9901b738-6f94-48a9-9282-a051ce9d37ae\") " pod="openshift-multus/multus-ht722"
Aug 02 09:32:58.917128 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.915320    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9901b738-6f94-48a9-9282-a051ce9d37ae-host-run-multus-certs\") pod \"multus-ht722\" (UID: \"9901b738-6f94-48a9-9282-a051ce9d37ae\") " pod="openshift-multus/multus-ht722"
Aug 02 09:32:58.917128 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.915344    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9901b738-6f94-48a9-9282-a051ce9d37ae-multus-conf-dir\") pod \"multus-ht722\" (UID: \"9901b738-6f94-48a9-9282-a051ce9d37ae\") " pod="openshift-multus/multus-ht722"
Aug 02 09:32:58.917128 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.915364    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9901b738-6f94-48a9-9282-a051ce9d37ae-host-var-lib-cni-multus\") pod \"multus-ht722\" (UID: \"9901b738-6f94-48a9-9282-a051ce9d37ae\") " pod="openshift-multus/multus-ht722"
Aug 02 09:32:58.917128 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.915376    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0fb6399d-1991-4b8e-920c-a1086603b643-cnibin\") pod \"multus-additional-cni-plugins-qgz2q\" (UID: \"0fb6399d-1991-4b8e-920c-a1086603b643\") " pod="openshift-multus/multus-additional-cni-plugins-qgz2q"
Aug 02 09:32:58.917128 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.915368    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3f97d5c9-1a25-43c5-8050-9c619be890e4-agent-certs\") pod \"konnectivity-agent-fhzz9\" (UID: \"3f97d5c9-1a25-43c5-8050-9c619be890e4\") " pod="kube-system/konnectivity-agent-fhzz9"
Aug 02 09:32:58.917128 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.915426    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6ddc3ad8-170f-49b5-8a1c-6fe966dd658a-socket-dir\") pod \"aws-ebs-csi-driver-node-tfhxd\" (UID: \"6ddc3ad8-170f-49b5-8a1c-6fe966dd658a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfhxd"
Aug 02 09:32:58.917128 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:32:58.915431    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 02 09:32:58.917128 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.915452    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9901b738-6f94-48a9-9282-a051ce9d37ae-host-run-netns\") pod \"multus-ht722\" (UID: \"9901b738-6f94-48a9-9282-a051ce9d37ae\") " pod="openshift-multus/multus-ht722"
Aug 02 09:32:58.917128 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:32:58.915495    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6944ae2-aaed-4a4a-9c46-dda99772fa54-original-pull-secret podName:e6944ae2-aaed-4a4a-9c46-dda99772fa54 nodeName:}" failed. No retries permitted until 2026-08-02 09:32:59.41547482 +0000 UTC m=+3.130280254 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e6944ae2-aaed-4a4a-9c46-dda99772fa54-original-pull-secret") pod "global-pull-secret-syncer-cs5vh" (UID: "e6944ae2-aaed-4a4a-9c46-dda99772fa54") : object "kube-system"/"original-pull-secret" not registered
Aug 02 09:32:58.917128 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.915500    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6ddc3ad8-170f-49b5-8a1c-6fe966dd658a-etc-selinux\") pod \"aws-ebs-csi-driver-node-tfhxd\" (UID: \"6ddc3ad8-170f-49b5-8a1c-6fe966dd658a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfhxd"
Aug 02 09:32:58.917128 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.915518    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6ddc3ad8-170f-49b5-8a1c-6fe966dd658a-sys-fs\") pod \"aws-ebs-csi-driver-node-tfhxd\" (UID: \"6ddc3ad8-170f-49b5-8a1c-6fe966dd658a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfhxd"
Aug 02 09:32:58.917128 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.915545    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9901b738-6f94-48a9-9282-a051ce9d37ae-multus-socket-dir-parent\") pod \"multus-ht722\" (UID: \"9901b738-6f94-48a9-9282-a051ce9d37ae\") " pod="openshift-multus/multus-ht722"
Aug 02 09:32:58.917874 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.915549    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e6944ae2-aaed-4a4a-9c46-dda99772fa54-kubelet-config\") pod \"global-pull-secret-syncer-cs5vh\" (UID: \"e6944ae2-aaed-4a4a-9c46-dda99772fa54\") " pod="kube-system/global-pull-secret-syncer-cs5vh"
Aug 02 09:32:58.917874 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.915546    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9901b738-6f94-48a9-9282-a051ce9d37ae-host-var-lib-cni-bin\") pod \"multus-ht722\" (UID: \"9901b738-6f94-48a9-9282-a051ce9d37ae\") " pod="openshift-multus/multus-ht722"
Aug 02 09:32:58.917874 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.915502    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9901b738-6f94-48a9-9282-a051ce9d37ae-host-run-netns\") pod \"multus-ht722\" (UID: \"9901b738-6f94-48a9-9282-a051ce9d37ae\") " pod="openshift-multus/multus-ht722"
Aug 02 09:32:58.917874 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.915599    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9901b738-6f94-48a9-9282-a051ce9d37ae-host-run-k8s-cni-cncf-io\") pod \"multus-ht722\" (UID: \"9901b738-6f94-48a9-9282-a051ce9d37ae\") " pod="openshift-multus/multus-ht722"
Aug 02 09:32:58.917874 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.915601    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0fb6399d-1991-4b8e-920c-a1086603b643-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qgz2q\" (UID: \"0fb6399d-1991-4b8e-920c-a1086603b643\") " pod="openshift-multus/multus-additional-cni-plugins-qgz2q"
Aug 02 09:32:58.917874 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.915638    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mb2c\" (UniqueName: \"kubernetes.io/projected/0fb6399d-1991-4b8e-920c-a1086603b643-kube-api-access-4mb2c\") pod \"multus-additional-cni-plugins-qgz2q\" (UID: \"0fb6399d-1991-4b8e-920c-a1086603b643\") " pod="openshift-multus/multus-additional-cni-plugins-qgz2q"
Aug 02 09:32:58.917874 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.915660    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6ddc3ad8-170f-49b5-8a1c-6fe966dd658a-socket-dir\") pod \"aws-ebs-csi-driver-node-tfhxd\" (UID: \"6ddc3ad8-170f-49b5-8a1c-6fe966dd658a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfhxd"
Aug 02 09:32:58.917874 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.915683    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2e5695b0-d7f7-4770-93a8-b9bcc79efad1-iptables-alerter-script\") pod \"iptables-alerter-878zb\" (UID: \"2e5695b0-d7f7-4770-93a8-b9bcc79efad1\") " pod="openshift-network-operator/iptables-alerter-878zb"
Aug 02 09:32:58.917874 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.915709    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e6944ae2-aaed-4a4a-9c46-dda99772fa54-dbus\") pod \"global-pull-secret-syncer-cs5vh\" (UID: \"e6944ae2-aaed-4a4a-9c46-dda99772fa54\") " pod="kube-system/global-pull-secret-syncer-cs5vh"
Aug 02 09:32:58.917874 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.915733    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9901b738-6f94-48a9-9282-a051ce9d37ae-cnibin\") pod \"multus-ht722\" (UID: \"9901b738-6f94-48a9-9282-a051ce9d37ae\") " pod="openshift-multus/multus-ht722"
Aug 02 09:32:58.917874 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.915753    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0fb6399d-1991-4b8e-920c-a1086603b643-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qgz2q\" (UID: \"0fb6399d-1991-4b8e-920c-a1086603b643\") " pod="openshift-multus/multus-additional-cni-plugins-qgz2q"
Aug 02 09:32:58.917874 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.915765    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl4s2\" (UniqueName: \"kubernetes.io/projected/2e5695b0-d7f7-4770-93a8-b9bcc79efad1-kube-api-access-wl4s2\") pod \"iptables-alerter-878zb\" (UID: \"2e5695b0-d7f7-4770-93a8-b9bcc79efad1\") " pod="openshift-network-operator/iptables-alerter-878zb"
Aug 02 09:32:58.917874 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.915791    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9901b738-6f94-48a9-9282-a051ce9d37ae-os-release\") pod \"multus-ht722\" (UID: \"9901b738-6f94-48a9-9282-a051ce9d37ae\") " pod="openshift-multus/multus-ht722"
Aug 02 09:32:58.917874 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.915812    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e6944ae2-aaed-4a4a-9c46-dda99772fa54-kubelet-config\") pod \"global-pull-secret-syncer-cs5vh\" (UID: \"e6944ae2-aaed-4a4a-9c46-dda99772fa54\") " pod="kube-system/global-pull-secret-syncer-cs5vh"
Aug 02 09:32:58.917874 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.915817    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6ddc3ad8-170f-49b5-8a1c-6fe966dd658a-sys-fs\") pod \"aws-ebs-csi-driver-node-tfhxd\" (UID: \"6ddc3ad8-170f-49b5-8a1c-6fe966dd658a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfhxd"
Aug 02 09:32:58.917874 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.915819    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6ddc3ad8-170f-49b5-8a1c-6fe966dd658a-registration-dir\") pod \"aws-ebs-csi-driver-node-tfhxd\" (UID: \"6ddc3ad8-170f-49b5-8a1c-6fe966dd658a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfhxd"
Aug 02 09:32:58.917874 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.915865    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9901b738-6f94-48a9-9282-a051ce9d37ae-cnibin\") pod \"multus-ht722\" (UID: \"9901b738-6f94-48a9-9282-a051ce9d37ae\") " pod="openshift-multus/multus-ht722"
Aug 02 09:32:58.918684 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.915872    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9901b738-6f94-48a9-9282-a051ce9d37ae-system-cni-dir\") pod \"multus-ht722\" (UID: \"9901b738-6f94-48a9-9282-a051ce9d37ae\") " pod="openshift-multus/multus-ht722"
Aug 02 09:32:58.918684 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.915878    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9901b738-6f94-48a9-9282-a051ce9d37ae-host-var-lib-kubelet\") pod \"multus-ht722\" (UID: \"9901b738-6f94-48a9-9282-a051ce9d37ae\") " pod="openshift-multus/multus-ht722"
Aug 02 09:32:58.918684 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.915887    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9901b738-6f94-48a9-9282-a051ce9d37ae-hostroot\") pod \"multus-ht722\" (UID: \"9901b738-6f94-48a9-9282-a051ce9d37ae\") " pod="openshift-multus/multus-ht722"
Aug 02 09:32:58.918684 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.915898    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3f97d5c9-1a25-43c5-8050-9c619be890e4-konnectivity-ca\") pod \"konnectivity-agent-fhzz9\" (UID: \"3f97d5c9-1a25-43c5-8050-9c619be890e4\") " pod="kube-system/konnectivity-agent-fhzz9"
Aug 02 09:32:58.918684 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.915910    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6ddc3ad8-170f-49b5-8a1c-6fe966dd658a-registration-dir\") pod \"aws-ebs-csi-driver-node-tfhxd\" (UID: \"6ddc3ad8-170f-49b5-8a1c-6fe966dd658a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfhxd"
Aug 02 09:32:58.918684 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.915953    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9901b738-6f94-48a9-9282-a051ce9d37ae-os-release\") pod \"multus-ht722\" (UID: \"9901b738-6f94-48a9-9282-a051ce9d37ae\") " pod="openshift-multus/multus-ht722"
Aug 02 09:32:58.918684 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.915988    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9901b738-6f94-48a9-9282-a051ce9d37ae-multus-conf-dir\") pod \"multus-ht722\" (UID: \"9901b738-6f94-48a9-9282-a051ce9d37ae\") " pod="openshift-multus/multus-ht722"
Aug 02 09:32:58.918684 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.916026    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtx9x\" (UniqueName: \"kubernetes.io/projected/3c47caf2-5d96-4343-ab9a-5da7195e209c-kube-api-access-mtx9x\") pod \"network-check-target-m9jmd\" (UID: \"3c47caf2-5d96-4343-ab9a-5da7195e209c\") " pod="openshift-network-diagnostics/network-check-target-m9jmd"
Aug 02 09:32:58.918684 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.916038    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e6944ae2-aaed-4a4a-9c46-dda99772fa54-dbus\") pod \"global-pull-secret-syncer-cs5vh\" (UID: \"e6944ae2-aaed-4a4a-9c46-dda99772fa54\") " pod="kube-system/global-pull-secret-syncer-cs5vh"
Aug 02 09:32:58.918684 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.916415    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0fb6399d-1991-4b8e-920c-a1086603b643-os-release\") pod \"multus-additional-cni-plugins-qgz2q\" (UID: \"0fb6399d-1991-4b8e-920c-a1086603b643\") " pod="openshift-multus/multus-additional-cni-plugins-qgz2q"
Aug 02 09:32:58.918684 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.916464    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2e5695b0-d7f7-4770-93a8-b9bcc79efad1-host-slash\") pod \"iptables-alerter-878zb\" (UID: \"2e5695b0-d7f7-4770-93a8-b9bcc79efad1\") " pod="openshift-network-operator/iptables-alerter-878zb"
Aug 02 09:32:58.918684 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.916502    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9901b738-6f94-48a9-9282-a051ce9d37ae-multus-cni-dir\") pod \"multus-ht722\" (UID: \"9901b738-6f94-48a9-9282-a051ce9d37ae\") " pod="openshift-multus/multus-ht722"
Aug 02 09:32:58.918684 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.916525    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9901b738-6f94-48a9-9282-a051ce9d37ae-multus-daemon-config\") pod \"multus-ht722\" (UID: \"9901b738-6f94-48a9-9282-a051ce9d37ae\") " pod="openshift-multus/multus-ht722"
Aug 02 09:32:58.918684 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.916574    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2e5695b0-d7f7-4770-93a8-b9bcc79efad1-iptables-alerter-script\") pod \"iptables-alerter-878zb\" (UID: \"2e5695b0-d7f7-4770-93a8-b9bcc79efad1\") " pod="openshift-network-operator/iptables-alerter-878zb"
Aug 02 09:32:58.918684 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.916591    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0fb6399d-1991-4b8e-920c-a1086603b643-os-release\") pod \"multus-additional-cni-plugins-qgz2q\" (UID: \"0fb6399d-1991-4b8e-920c-a1086603b643\") " pod="openshift-multus/multus-additional-cni-plugins-qgz2q"
Aug 02 09:32:58.918684 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.916591    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9901b738-6f94-48a9-9282-a051ce9d37ae-multus-cni-dir\") pod \"multus-ht722\" (UID: \"9901b738-6f94-48a9-9282-a051ce9d37ae\") " pod="openshift-multus/multus-ht722"
Aug 02 09:32:58.918684 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.916616    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9901b738-6f94-48a9-9282-a051ce9d37ae-etc-kubernetes\") pod \"multus-ht722\" (UID: \"9901b738-6f94-48a9-9282-a051ce9d37ae\") " pod="openshift-multus/multus-ht722"
Aug 02 09:32:58.918684 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.916639    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2e5695b0-d7f7-4770-93a8-b9bcc79efad1-host-slash\") pod \"iptables-alerter-878zb\" (UID: \"2e5695b0-d7f7-4770-93a8-b9bcc79efad1\") " pod="openshift-network-operator/iptables-alerter-878zb"
Aug 02 09:32:58.919664 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.916655    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0fb6399d-1991-4b8e-920c-a1086603b643-cni-binary-copy\") pod \"multus-additional-cni-plugins-qgz2q\" (UID: \"0fb6399d-1991-4b8e-920c-a1086603b643\") " pod="openshift-multus/multus-additional-cni-plugins-qgz2q"
Aug 02 09:32:58.919664 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.916674    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mjgs\" (UniqueName: \"kubernetes.io/projected/9901b738-6f94-48a9-9282-a051ce9d37ae-kube-api-access-5mjgs\") pod \"multus-ht722\" (UID: \"9901b738-6f94-48a9-9282-a051ce9d37ae\") " pod="openshift-multus/multus-ht722"
Aug 02 09:32:58.919664 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.916702    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9901b738-6f94-48a9-9282-a051ce9d37ae-etc-kubernetes\") pod \"multus-ht722\" (UID: \"9901b738-6f94-48a9-9282-a051ce9d37ae\") " pod="openshift-multus/multus-ht722"
Aug 02 09:32:58.919664 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.916709    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0fb6399d-1991-4b8e-920c-a1086603b643-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qgz2q\" (UID: \"0fb6399d-1991-4b8e-920c-a1086603b643\") " pod="openshift-multus/multus-additional-cni-plugins-qgz2q"
Aug 02 09:32:58.919664 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.916655    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3f97d5c9-1a25-43c5-8050-9c619be890e4-konnectivity-ca\") pod \"konnectivity-agent-fhzz9\" (UID: \"3f97d5c9-1a25-43c5-8050-9c619be890e4\") " pod="kube-system/konnectivity-agent-fhzz9"
Aug 02 09:32:58.919664 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.916738    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6ddc3ad8-170f-49b5-8a1c-6fe966dd658a-device-dir\") pod \"aws-ebs-csi-driver-node-tfhxd\" (UID: \"6ddc3ad8-170f-49b5-8a1c-6fe966dd658a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfhxd"
Aug 02 09:32:58.919664 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.916764    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9901b738-6f94-48a9-9282-a051ce9d37ae-cni-binary-copy\") pod \"multus-ht722\" (UID: \"9901b738-6f94-48a9-9282-a051ce9d37ae\") " pod="openshift-multus/multus-ht722"
Aug 02 09:32:58.919664 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.916788    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0fb6399d-1991-4b8e-920c-a1086603b643-system-cni-dir\") pod \"multus-additional-cni-plugins-qgz2q\" (UID: \"0fb6399d-1991-4b8e-920c-a1086603b643\") " pod="openshift-multus/multus-additional-cni-plugins-qgz2q"
Aug 02 09:32:58.919664 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.916814    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0fb6399d-1991-4b8e-920c-a1086603b643-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qgz2q\" (UID: \"0fb6399d-1991-4b8e-920c-a1086603b643\") " pod="openshift-multus/multus-additional-cni-plugins-qgz2q"
Aug 02 09:32:58.919664 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.916839    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6ddc3ad8-170f-49b5-8a1c-6fe966dd658a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tfhxd\" (UID: \"6ddc3ad8-170f-49b5-8a1c-6fe966dd658a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfhxd"
Aug 02 09:32:58.919664 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.916845    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6ddc3ad8-170f-49b5-8a1c-6fe966dd658a-device-dir\") pod \"aws-ebs-csi-driver-node-tfhxd\" (UID: \"6ddc3ad8-170f-49b5-8a1c-6fe966dd658a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfhxd"
Aug 02 09:32:58.919664 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.916904    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0fb6399d-1991-4b8e-920c-a1086603b643-system-cni-dir\") pod \"multus-additional-cni-plugins-qgz2q\" (UID: \"0fb6399d-1991-4b8e-920c-a1086603b643\") " pod="openshift-multus/multus-additional-cni-plugins-qgz2q"
Aug 02 09:32:58.919664 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.916966    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6ddc3ad8-170f-49b5-8a1c-6fe966dd658a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tfhxd\" (UID: \"6ddc3ad8-170f-49b5-8a1c-6fe966dd658a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfhxd"
Aug 02 09:32:58.919664 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.917189    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9901b738-6f94-48a9-9282-a051ce9d37ae-system-cni-dir\") pod \"multus-ht722\" (UID: \"9901b738-6f94-48a9-9282-a051ce9d37ae\") " pod="openshift-multus/multus-ht722"
Aug 02 09:32:58.919664 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.917373    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0fb6399d-1991-4b8e-920c-a1086603b643-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qgz2q\" (UID: \"0fb6399d-1991-4b8e-920c-a1086603b643\") " pod="openshift-multus/multus-additional-cni-plugins-qgz2q"
Aug 02 09:32:58.919664 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.917412    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9901b738-6f94-48a9-9282-a051ce9d37ae-cni-binary-copy\") pod \"multus-ht722\" (UID: \"9901b738-6f94-48a9-9282-a051ce9d37ae\") " pod="openshift-multus/multus-ht722"
Aug 02 09:32:58.919664 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.917420    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0fb6399d-1991-4b8e-920c-a1086603b643-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qgz2q\" (UID: \"0fb6399d-1991-4b8e-920c-a1086603b643\") " pod="openshift-multus/multus-additional-cni-plugins-qgz2q"
Aug 02 09:32:58.920399 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.917900    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9901b738-6f94-48a9-9282-a051ce9d37ae-multus-daemon-config\") pod \"multus-ht722\" (UID: \"9901b738-6f94-48a9-9282-a051ce9d37ae\") " pod="openshift-multus/multus-ht722"
Aug 02 09:32:58.920399 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.918565    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3f97d5c9-1a25-43c5-8050-9c619be890e4-agent-certs\") pod \"konnectivity-agent-fhzz9\" (UID: \"3f97d5c9-1a25-43c5-8050-9c619be890e4\") " pod="kube-system/konnectivity-agent-fhzz9"
Aug 02 09:32:58.923452 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:32:58.923429    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 02 09:32:58.923452 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:32:58.923455    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 02 09:32:58.923624 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:32:58.923470    2564 projected.go:196] Error preparing data for projected volume kube-api-access-mtx9x for pod openshift-network-diagnostics/network-check-target-m9jmd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 02 09:32:58.923624 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:32:58.923527    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c47caf2-5d96-4343-ab9a-5da7195e209c-kube-api-access-mtx9x podName:3c47caf2-5d96-4343-ab9a-5da7195e209c nodeName:}" failed. No retries permitted until 2026-08-02 09:32:59.423509489 +0000 UTC m=+3.138314937 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mtx9x" (UniqueName: "kubernetes.io/projected/3c47caf2-5d96-4343-ab9a-5da7195e209c-kube-api-access-mtx9x") pod "network-check-target-m9jmd" (UID: "3c47caf2-5d96-4343-ab9a-5da7195e209c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 02 09:32:58.925359 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.925335    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl4s2\" (UniqueName: \"kubernetes.io/projected/2e5695b0-d7f7-4770-93a8-b9bcc79efad1-kube-api-access-wl4s2\") pod \"iptables-alerter-878zb\" (UID: \"2e5695b0-d7f7-4770-93a8-b9bcc79efad1\") " pod="openshift-network-operator/iptables-alerter-878zb"
Aug 02 09:32:58.925472 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.925360    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mjgs\" (UniqueName: \"kubernetes.io/projected/9901b738-6f94-48a9-9282-a051ce9d37ae-kube-api-access-5mjgs\") pod \"multus-ht722\" (UID: \"9901b738-6f94-48a9-9282-a051ce9d37ae\") " pod="openshift-multus/multus-ht722"
Aug 02 09:32:58.926284 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.926236    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-842wd\" (UniqueName: \"kubernetes.io/projected/6ddc3ad8-170f-49b5-8a1c-6fe966dd658a-kube-api-access-842wd\") pod \"aws-ebs-csi-driver-node-tfhxd\" (UID: \"6ddc3ad8-170f-49b5-8a1c-6fe966dd658a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfhxd"
Aug 02 09:32:58.926383 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:58.926325    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mb2c\" (UniqueName: \"kubernetes.io/projected/0fb6399d-1991-4b8e-920c-a1086603b643-kube-api-access-4mb2c\") pod \"multus-additional-cni-plugins-qgz2q\" (UID: \"0fb6399d-1991-4b8e-920c-a1086603b643\") " pod="openshift-multus/multus-additional-cni-plugins-qgz2q"
Aug 02 09:32:59.013619 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:59.013546    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8lm5w"
Aug 02 09:32:59.021650 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:59.021628    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fj9zj"
Aug 02 09:32:59.029523 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:59.029498    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4f6pd"
Aug 02 09:32:59.038338 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:59.038296    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:32:59.047193 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:59.047172    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qgz2q"
Aug 02 09:32:59.056042 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:59.056022    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fhzz9"
Aug 02 09:32:59.063799 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:59.063781    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-878zb"
Aug 02 09:32:59.074527 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:59.074508    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfhxd"
Aug 02 09:32:59.081420 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:59.081398    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ht722"
Aug 02 09:32:59.086942 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:59.086923    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 02 09:32:59.319632 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:59.319550    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2068f737-ee2f-4310-977e-27cb7e1d4f73-metrics-certs\") pod \"network-metrics-daemon-v5mrn\" (UID: \"2068f737-ee2f-4310-977e-27cb7e1d4f73\") " pod="openshift-multus/network-metrics-daemon-v5mrn"
Aug 02 09:32:59.319799 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:32:59.319730    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 02 09:32:59.319872 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:32:59.319806    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2068f737-ee2f-4310-977e-27cb7e1d4f73-metrics-certs podName:2068f737-ee2f-4310-977e-27cb7e1d4f73 nodeName:}" failed. No retries permitted until 2026-08-02 09:33:00.319784411 +0000 UTC m=+4.034589857 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2068f737-ee2f-4310-977e-27cb7e1d4f73-metrics-certs") pod "network-metrics-daemon-v5mrn" (UID: "2068f737-ee2f-4310-977e-27cb7e1d4f73") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 02 09:32:59.420577 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:59.420546    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e6944ae2-aaed-4a4a-9c46-dda99772fa54-original-pull-secret\") pod \"global-pull-secret-syncer-cs5vh\" (UID: \"e6944ae2-aaed-4a4a-9c46-dda99772fa54\") " pod="kube-system/global-pull-secret-syncer-cs5vh"
Aug 02 09:32:59.420748 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:32:59.420712    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 02 09:32:59.420805 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:32:59.420780    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6944ae2-aaed-4a4a-9c46-dda99772fa54-original-pull-secret podName:e6944ae2-aaed-4a4a-9c46-dda99772fa54 nodeName:}" failed. No retries permitted until 2026-08-02 09:33:00.420763117 +0000 UTC m=+4.135568545 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e6944ae2-aaed-4a4a-9c46-dda99772fa54-original-pull-secret") pod "global-pull-secret-syncer-cs5vh" (UID: "e6944ae2-aaed-4a4a-9c46-dda99772fa54") : object "kube-system"/"original-pull-secret" not registered
Aug 02 09:32:59.462967 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:59.462936    2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 02 09:32:59.521626 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:59.521587    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtx9x\" (UniqueName: \"kubernetes.io/projected/3c47caf2-5d96-4343-ab9a-5da7195e209c-kube-api-access-mtx9x\") pod \"network-check-target-m9jmd\" (UID: \"3c47caf2-5d96-4343-ab9a-5da7195e209c\") " pod="openshift-network-diagnostics/network-check-target-m9jmd"
Aug 02 09:32:59.521809 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:32:59.521787    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 02 09:32:59.521881 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:32:59.521816    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 02 09:32:59.521881 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:32:59.521827    2564 projected.go:196] Error preparing data for projected volume kube-api-access-mtx9x for pod openshift-network-diagnostics/network-check-target-m9jmd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 02 09:32:59.521981 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:32:59.521890    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c47caf2-5d96-4343-ab9a-5da7195e209c-kube-api-access-mtx9x podName:3c47caf2-5d96-4343-ab9a-5da7195e209c nodeName:}" failed. No retries permitted until 2026-08-02 09:33:00.521868992 +0000 UTC m=+4.236674452 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mtx9x" (UniqueName: "kubernetes.io/projected/3c47caf2-5d96-4343-ab9a-5da7195e209c-kube-api-access-mtx9x") pod "network-check-target-m9jmd" (UID: "3c47caf2-5d96-4343-ab9a-5da7195e209c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 02 09:32:59.737158 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:59.737044    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-01 09:27:59 +0000 UTC" deadline="2028-02-09 15:39:43.292393985 +0000 UTC"
Aug 02 09:32:59.737158 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:59.737102    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13350h6m43.555297038s"
Aug 02 09:32:59.875268 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:32:59.875201    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cs5vh"
Aug 02 09:32:59.875439 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:32:59.875385    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cs5vh" podUID="e6944ae2-aaed-4a4a-9c46-dda99772fa54"
Aug 02 09:33:00.052717 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:33:00.052676    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95220006_a44c_46f9_b380_f771acbd1eac.slice/crio-7908004b00c083c5d5cbd2e6fa3c9563b04e007203f95db69d95ce0703e1dcd6 WatchSource:0}: Error finding container 7908004b00c083c5d5cbd2e6fa3c9563b04e007203f95db69d95ce0703e1dcd6: Status 404 returned error can't find the container with id 7908004b00c083c5d5cbd2e6fa3c9563b04e007203f95db69d95ce0703e1dcd6
Aug 02 09:33:00.054085 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:33:00.054041    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9901b738_6f94_48a9_9282_a051ce9d37ae.slice/crio-c14d21cfff82762e321bb3fdbb8121017842060df10d9206db97e74aef1968b1 WatchSource:0}: Error finding container c14d21cfff82762e321bb3fdbb8121017842060df10d9206db97e74aef1968b1: Status 404 returned error can't find the container with id c14d21cfff82762e321bb3fdbb8121017842060df10d9206db97e74aef1968b1
Aug 02 09:33:00.054925 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:33:00.054899    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b0cc25e_0367_4ac6_b094_1caf4d2023ab.slice/crio-09b81633a57000f44e5cb9fc0ac5e9689d44d79477e15d8ae4e026cfd97c7b32 WatchSource:0}: Error finding container 09b81633a57000f44e5cb9fc0ac5e9689d44d79477e15d8ae4e026cfd97c7b32: Status 404 returned error can't find the container with id 09b81633a57000f44e5cb9fc0ac5e9689d44d79477e15d8ae4e026cfd97c7b32
Aug 02 09:33:00.055862 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:33:00.055838    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90afa8c6_44be_4641_aa6b_90b51f5c1c6b.slice/crio-fd678161a7bdc7c1e9f7774cd5aee4e5597195bdbcac797b481a86d61a6f97e9 WatchSource:0}: Error finding container fd678161a7bdc7c1e9f7774cd5aee4e5597195bdbcac797b481a86d61a6f97e9: Status 404 returned error can't find the container with id fd678161a7bdc7c1e9f7774cd5aee4e5597195bdbcac797b481a86d61a6f97e9
Aug 02 09:33:00.058338 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:33:00.058318    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f97d5c9_1a25_43c5_8050_9c619be890e4.slice/crio-1d2b447dd7d3624d299666434340e8d2332afc49e75caeb62e7503ab55e36ef5 WatchSource:0}: Error finding container 1d2b447dd7d3624d299666434340e8d2332afc49e75caeb62e7503ab55e36ef5: Status 404 returned error can't find the container with id 1d2b447dd7d3624d299666434340e8d2332afc49e75caeb62e7503ab55e36ef5
Aug 02 09:33:00.082425 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:33:00.082399    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fb6399d_1991_4b8e_920c_a1086603b643.slice/crio-543eaaa450c068a87dc9f8504072c77b20cd78319e04e3edaf5b870f4bce87c4 WatchSource:0}: Error finding container 543eaaa450c068a87dc9f8504072c77b20cd78319e04e3edaf5b870f4bce87c4: Status 404 returned error can't find the container with id 543eaaa450c068a87dc9f8504072c77b20cd78319e04e3edaf5b870f4bce87c4
Aug 02 09:33:00.083008 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:33:00.082982    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ddc3ad8_170f_49b5_8a1c_6fe966dd658a.slice/crio-ce8fb72528c032ce157453b72a2d519fa644ba8a29fec423164cd39b71c4a96c WatchSource:0}: Error finding container ce8fb72528c032ce157453b72a2d519fa644ba8a29fec423164cd39b71c4a96c: Status 404 returned error can't find the container with id ce8fb72528c032ce157453b72a2d519fa644ba8a29fec423164cd39b71c4a96c
Aug 02 09:33:00.084490 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:33:00.084403    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e5695b0_d7f7_4770_93a8_b9bcc79efad1.slice/crio-cd74f1cc3e7cfabab895742c76a515d2219bf0370b4dd91d1c118aea8034bd12 WatchSource:0}: Error finding container cd74f1cc3e7cfabab895742c76a515d2219bf0370b4dd91d1c118aea8034bd12: Status 404 returned error can't find the container with id cd74f1cc3e7cfabab895742c76a515d2219bf0370b4dd91d1c118aea8034bd12
Aug 02 09:33:00.326297 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:00.326124    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2068f737-ee2f-4310-977e-27cb7e1d4f73-metrics-certs\") pod \"network-metrics-daemon-v5mrn\" (UID: \"2068f737-ee2f-4310-977e-27cb7e1d4f73\") " pod="openshift-multus/network-metrics-daemon-v5mrn"
Aug 02 09:33:00.326297 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:00.326288    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 02 09:33:00.326471 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:00.326364    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2068f737-ee2f-4310-977e-27cb7e1d4f73-metrics-certs podName:2068f737-ee2f-4310-977e-27cb7e1d4f73 nodeName:}" failed. No retries permitted until 2026-08-02 09:33:02.326345582 +0000 UTC m=+6.041151024 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2068f737-ee2f-4310-977e-27cb7e1d4f73-metrics-certs") pod "network-metrics-daemon-v5mrn" (UID: "2068f737-ee2f-4310-977e-27cb7e1d4f73") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 02 09:33:00.427629 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:00.427603    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e6944ae2-aaed-4a4a-9c46-dda99772fa54-original-pull-secret\") pod \"global-pull-secret-syncer-cs5vh\" (UID: \"e6944ae2-aaed-4a4a-9c46-dda99772fa54\") " pod="kube-system/global-pull-secret-syncer-cs5vh"
Aug 02 09:33:00.427733 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:00.427721    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 02 09:33:00.427782 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:00.427773    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6944ae2-aaed-4a4a-9c46-dda99772fa54-original-pull-secret podName:e6944ae2-aaed-4a4a-9c46-dda99772fa54 nodeName:}" failed. No retries permitted until 2026-08-02 09:33:02.427759216 +0000 UTC m=+6.142564643 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e6944ae2-aaed-4a4a-9c46-dda99772fa54-original-pull-secret") pod "global-pull-secret-syncer-cs5vh" (UID: "e6944ae2-aaed-4a4a-9c46-dda99772fa54") : object "kube-system"/"original-pull-secret" not registered
Aug 02 09:33:00.528691 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:00.528656    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtx9x\" (UniqueName: \"kubernetes.io/projected/3c47caf2-5d96-4343-ab9a-5da7195e209c-kube-api-access-mtx9x\") pod \"network-check-target-m9jmd\" (UID: \"3c47caf2-5d96-4343-ab9a-5da7195e209c\") " pod="openshift-network-diagnostics/network-check-target-m9jmd"
Aug 02 09:33:00.528929 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:00.528824    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 02 09:33:00.528929 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:00.528847    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 02 09:33:00.528929 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:00.528861    2564 projected.go:196] Error preparing data for projected volume kube-api-access-mtx9x for pod openshift-network-diagnostics/network-check-target-m9jmd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 02 09:33:00.528929 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:00.528927    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c47caf2-5d96-4343-ab9a-5da7195e209c-kube-api-access-mtx9x podName:3c47caf2-5d96-4343-ab9a-5da7195e209c nodeName:}" failed. No retries permitted until 2026-08-02 09:33:02.528908878 +0000 UTC m=+6.243714313 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mtx9x" (UniqueName: "kubernetes.io/projected/3c47caf2-5d96-4343-ab9a-5da7195e209c-kube-api-access-mtx9x") pod "network-check-target-m9jmd" (UID: "3c47caf2-5d96-4343-ab9a-5da7195e209c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 02 09:33:00.737759 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:00.737590    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-01 09:27:59 +0000 UTC" deadline="2028-04-20 12:55:10.80075428 +0000 UTC"
Aug 02 09:33:00.737759 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:00.737644    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15051h22m10.063114371s"
Aug 02 09:33:00.889019 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:00.888161    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5mrn"
Aug 02 09:33:00.889019 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:00.888328    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5mrn" podUID="2068f737-ee2f-4310-977e-27cb7e1d4f73"
Aug 02 09:33:00.889019 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:00.888737    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jmd"
Aug 02 09:33:00.889019 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:00.888835    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jmd" podUID="3c47caf2-5d96-4343-ab9a-5da7195e209c"
Aug 02 09:33:00.906040 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:00.906010    2564 generic.go:356] "Generic (PLEG): container finished" podID="5fa2085f44de3c0708af14f57a9f26c5" containerID="23fc71c993ed627ecb9c089f95e5373ee57f4d4a6bf1e33d9a507d89835e0635" exitCode=0
Aug 02 09:33:00.906186 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:00.906093    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal" event={"ID":"5fa2085f44de3c0708af14f57a9f26c5","Type":"ContainerDied","Data":"23fc71c993ed627ecb9c089f95e5373ee57f4d4a6bf1e33d9a507d89835e0635"}
Aug 02 09:33:00.909177 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:00.909147    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-878zb" event={"ID":"2e5695b0-d7f7-4770-93a8-b9bcc79efad1","Type":"ContainerStarted","Data":"cd74f1cc3e7cfabab895742c76a515d2219bf0370b4dd91d1c118aea8034bd12"}
Aug 02 09:33:00.915641 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:00.915615    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ht722" event={"ID":"9901b738-6f94-48a9-9282-a051ce9d37ae","Type":"ContainerStarted","Data":"c14d21cfff82762e321bb3fdbb8121017842060df10d9206db97e74aef1968b1"}
Aug 02 09:33:00.924399 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:00.924335    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8lm5w" event={"ID":"6b0cc25e-0367-4ac6-b094-1caf4d2023ab","Type":"ContainerStarted","Data":"09b81633a57000f44e5cb9fc0ac5e9689d44d79477e15d8ae4e026cfd97c7b32"}
Aug 02 09:33:00.939741 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:00.939685    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fj9zj" event={"ID":"95220006-a44c-46f9-b380-f771acbd1eac","Type":"ContainerStarted","Data":"7908004b00c083c5d5cbd2e6fa3c9563b04e007203f95db69d95ce0703e1dcd6"}
Aug 02 09:33:00.945911 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:00.945883    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-232.ec2.internal" event={"ID":"ce69cf0400f5bcb61dc1156a46d82c8f","Type":"ContainerStarted","Data":"4aa6f71ff3eccec912fafe5c578de83a5f146cfbb80ae6e05c57fdce819982e2"}
Aug 02 09:33:00.949745 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:00.949720    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4f6pd" event={"ID":"032156ef-c7a4-4cd5-a991-4c91f2a67b4b","Type":"ContainerStarted","Data":"eb3e4c22a406c8b3740c08aa5d96dcba89c62e7cfba78235cfcf2d7ac5d569bb"}
Aug 02 09:33:00.955108 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:00.955083    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qgz2q" event={"ID":"0fb6399d-1991-4b8e-920c-a1086603b643","Type":"ContainerStarted","Data":"543eaaa450c068a87dc9f8504072c77b20cd78319e04e3edaf5b870f4bce87c4"}
Aug 02 09:33:00.957550 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:00.957525    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfhxd" event={"ID":"6ddc3ad8-170f-49b5-8a1c-6fe966dd658a","Type":"ContainerStarted","Data":"ce8fb72528c032ce157453b72a2d519fa644ba8a29fec423164cd39b71c4a96c"}
Aug 02 09:33:00.974428 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:00.974399    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fhzz9" event={"ID":"3f97d5c9-1a25-43c5-8050-9c619be890e4","Type":"ContainerStarted","Data":"1d2b447dd7d3624d299666434340e8d2332afc49e75caeb62e7503ab55e36ef5"}
Aug 02 09:33:00.997405 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:00.993929    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-59x9w" event={"ID":"90afa8c6-44be-4641-aa6b-90b51f5c1c6b","Type":"ContainerStarted","Data":"fd678161a7bdc7c1e9f7774cd5aee4e5597195bdbcac797b481a86d61a6f97e9"}
Aug 02 09:33:01.875243 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:01.875155    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cs5vh"
Aug 02 09:33:01.875737 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:01.875320    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cs5vh" podUID="e6944ae2-aaed-4a4a-9c46-dda99772fa54"
Aug 02 09:33:02.024884 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:02.024805    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal" event={"ID":"5fa2085f44de3c0708af14f57a9f26c5","Type":"ContainerStarted","Data":"9d1dc0b913efd875cfd290d766d90f42557c99d6ca340d9ad3aa3951ce1d1928"}
Aug 02 09:33:02.040228 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:02.040144    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-232.ec2.internal" podStartSLOduration=4.040121025 podStartE2EDuration="4.040121025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-02 09:32:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:33:00.960347056 +0000 UTC m=+4.675152507" watchObservedRunningTime="2026-08-02 09:33:02.040121025 +0000 UTC m=+5.754926479"
Aug 02 09:33:02.041075 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:02.041030    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal" podStartSLOduration=4.04101332 podStartE2EDuration="4.04101332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-02 09:32:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:33:02.039761219 +0000 UTC m=+5.754566664" watchObservedRunningTime="2026-08-02 09:33:02.04101332 +0000 UTC m=+5.755818774"
Aug 02 09:33:02.348559 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:02.347806    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2068f737-ee2f-4310-977e-27cb7e1d4f73-metrics-certs\") pod \"network-metrics-daemon-v5mrn\" (UID: \"2068f737-ee2f-4310-977e-27cb7e1d4f73\") " pod="openshift-multus/network-metrics-daemon-v5mrn"
Aug 02 09:33:02.348559 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:02.347984    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 02 09:33:02.348559 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:02.348049    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2068f737-ee2f-4310-977e-27cb7e1d4f73-metrics-certs podName:2068f737-ee2f-4310-977e-27cb7e1d4f73 nodeName:}" failed. No retries permitted until 2026-08-02 09:33:06.34802776 +0000 UTC m=+10.062833191 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2068f737-ee2f-4310-977e-27cb7e1d4f73-metrics-certs") pod "network-metrics-daemon-v5mrn" (UID: "2068f737-ee2f-4310-977e-27cb7e1d4f73") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 02 09:33:02.449420 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:02.448792    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e6944ae2-aaed-4a4a-9c46-dda99772fa54-original-pull-secret\") pod \"global-pull-secret-syncer-cs5vh\" (UID: \"e6944ae2-aaed-4a4a-9c46-dda99772fa54\") " pod="kube-system/global-pull-secret-syncer-cs5vh"
Aug 02 09:33:02.449420 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:02.448933    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 02 09:33:02.449420 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:02.448994    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6944ae2-aaed-4a4a-9c46-dda99772fa54-original-pull-secret podName:e6944ae2-aaed-4a4a-9c46-dda99772fa54 nodeName:}" failed. No retries permitted until 2026-08-02 09:33:06.448975192 +0000 UTC m=+10.163780623 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e6944ae2-aaed-4a4a-9c46-dda99772fa54-original-pull-secret") pod "global-pull-secret-syncer-cs5vh" (UID: "e6944ae2-aaed-4a4a-9c46-dda99772fa54") : object "kube-system"/"original-pull-secret" not registered
Aug 02 09:33:02.550811 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:02.550110    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtx9x\" (UniqueName: \"kubernetes.io/projected/3c47caf2-5d96-4343-ab9a-5da7195e209c-kube-api-access-mtx9x\") pod \"network-check-target-m9jmd\" (UID: \"3c47caf2-5d96-4343-ab9a-5da7195e209c\") " pod="openshift-network-diagnostics/network-check-target-m9jmd"
Aug 02 09:33:02.550811 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:02.550341    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 02 09:33:02.550811 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:02.550365    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 02 09:33:02.550811 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:02.550385    2564 projected.go:196] Error preparing data for projected volume kube-api-access-mtx9x for pod openshift-network-diagnostics/network-check-target-m9jmd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 02 09:33:02.550811 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:02.550448    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c47caf2-5d96-4343-ab9a-5da7195e209c-kube-api-access-mtx9x podName:3c47caf2-5d96-4343-ab9a-5da7195e209c nodeName:}" failed. No retries permitted until 2026-08-02 09:33:06.550427953 +0000 UTC m=+10.265233404 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mtx9x" (UniqueName: "kubernetes.io/projected/3c47caf2-5d96-4343-ab9a-5da7195e209c-kube-api-access-mtx9x") pod "network-check-target-m9jmd" (UID: "3c47caf2-5d96-4343-ab9a-5da7195e209c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 02 09:33:02.877878 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:02.877848    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5mrn"
Aug 02 09:33:02.878406 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:02.877990    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5mrn" podUID="2068f737-ee2f-4310-977e-27cb7e1d4f73"
Aug 02 09:33:02.878406 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:02.878393    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jmd"
Aug 02 09:33:02.878536 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:02.878491    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jmd" podUID="3c47caf2-5d96-4343-ab9a-5da7195e209c"
Aug 02 09:33:03.875303 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:03.875267    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cs5vh"
Aug 02 09:33:03.875505 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:03.875415    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cs5vh" podUID="e6944ae2-aaed-4a4a-9c46-dda99772fa54"
Aug 02 09:33:04.876206 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:04.875537    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5mrn"
Aug 02 09:33:04.876206 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:04.875687    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5mrn" podUID="2068f737-ee2f-4310-977e-27cb7e1d4f73"
Aug 02 09:33:04.876206 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:04.876078    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jmd"
Aug 02 09:33:04.876206 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:04.876165    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jmd" podUID="3c47caf2-5d96-4343-ab9a-5da7195e209c"
Aug 02 09:33:05.875454 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:05.875418    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cs5vh"
Aug 02 09:33:05.875645 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:05.875567    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cs5vh" podUID="e6944ae2-aaed-4a4a-9c46-dda99772fa54"
Aug 02 09:33:06.384540 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:06.384389    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2068f737-ee2f-4310-977e-27cb7e1d4f73-metrics-certs\") pod \"network-metrics-daemon-v5mrn\" (UID: \"2068f737-ee2f-4310-977e-27cb7e1d4f73\") " pod="openshift-multus/network-metrics-daemon-v5mrn"
Aug 02 09:33:06.385024 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:06.384610    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 02 09:33:06.385024 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:06.384683    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2068f737-ee2f-4310-977e-27cb7e1d4f73-metrics-certs podName:2068f737-ee2f-4310-977e-27cb7e1d4f73 nodeName:}" failed. No retries permitted until 2026-08-02 09:33:14.384662027 +0000 UTC m=+18.099467455 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2068f737-ee2f-4310-977e-27cb7e1d4f73-metrics-certs") pod "network-metrics-daemon-v5mrn" (UID: "2068f737-ee2f-4310-977e-27cb7e1d4f73") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 02 09:33:06.485171 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:06.485139    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e6944ae2-aaed-4a4a-9c46-dda99772fa54-original-pull-secret\") pod \"global-pull-secret-syncer-cs5vh\" (UID: \"e6944ae2-aaed-4a4a-9c46-dda99772fa54\") " pod="kube-system/global-pull-secret-syncer-cs5vh"
Aug 02 09:33:06.485374 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:06.485316    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 02 09:33:06.485374 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:06.485369    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6944ae2-aaed-4a4a-9c46-dda99772fa54-original-pull-secret podName:e6944ae2-aaed-4a4a-9c46-dda99772fa54 nodeName:}" failed. No retries permitted until 2026-08-02 09:33:14.485352648 +0000 UTC m=+18.200158075 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e6944ae2-aaed-4a4a-9c46-dda99772fa54-original-pull-secret") pod "global-pull-secret-syncer-cs5vh" (UID: "e6944ae2-aaed-4a4a-9c46-dda99772fa54") : object "kube-system"/"original-pull-secret" not registered
Aug 02 09:33:06.586297 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:06.586259    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtx9x\" (UniqueName: \"kubernetes.io/projected/3c47caf2-5d96-4343-ab9a-5da7195e209c-kube-api-access-mtx9x\") pod \"network-check-target-m9jmd\" (UID: \"3c47caf2-5d96-4343-ab9a-5da7195e209c\") " pod="openshift-network-diagnostics/network-check-target-m9jmd"
Aug 02 09:33:06.586450 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:06.586424    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 02 09:33:06.586450 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:06.586446    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 02 09:33:06.586539 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:06.586459    2564 projected.go:196] Error preparing data for projected volume kube-api-access-mtx9x for pod openshift-network-diagnostics/network-check-target-m9jmd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 02 09:33:06.586539 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:06.586518    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c47caf2-5d96-4343-ab9a-5da7195e209c-kube-api-access-mtx9x podName:3c47caf2-5d96-4343-ab9a-5da7195e209c nodeName:}" failed. No retries permitted until 2026-08-02 09:33:14.586499115 +0000 UTC m=+18.301304542 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mtx9x" (UniqueName: "kubernetes.io/projected/3c47caf2-5d96-4343-ab9a-5da7195e209c-kube-api-access-mtx9x") pod "network-check-target-m9jmd" (UID: "3c47caf2-5d96-4343-ab9a-5da7195e209c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 02 09:33:06.876549 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:06.876465    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jmd"
Aug 02 09:33:06.876706 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:06.876569    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jmd" podUID="3c47caf2-5d96-4343-ab9a-5da7195e209c"
Aug 02 09:33:06.876706 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:06.876622    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5mrn"
Aug 02 09:33:06.876807 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:06.876754    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5mrn" podUID="2068f737-ee2f-4310-977e-27cb7e1d4f73"
Aug 02 09:33:07.875319 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:07.875278    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cs5vh"
Aug 02 09:33:07.875807 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:07.875414    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cs5vh" podUID="e6944ae2-aaed-4a4a-9c46-dda99772fa54"
Aug 02 09:33:08.875473 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:08.875441    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jmd"
Aug 02 09:33:08.875473 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:08.875473    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5mrn"
Aug 02 09:33:08.875984 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:08.875611    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5mrn" podUID="2068f737-ee2f-4310-977e-27cb7e1d4f73"
Aug 02 09:33:08.875984 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:08.875741    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jmd" podUID="3c47caf2-5d96-4343-ab9a-5da7195e209c"
Aug 02 09:33:09.874842 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:09.874805    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cs5vh"
Aug 02 09:33:09.875021 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:09.874930    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cs5vh" podUID="e6944ae2-aaed-4a4a-9c46-dda99772fa54"
Aug 02 09:33:10.875046 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:10.875008    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jmd"
Aug 02 09:33:10.875551 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:10.875139    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jmd" podUID="3c47caf2-5d96-4343-ab9a-5da7195e209c"
Aug 02 09:33:10.875551 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:10.875191    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5mrn"
Aug 02 09:33:10.875551 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:10.875334    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5mrn" podUID="2068f737-ee2f-4310-977e-27cb7e1d4f73"
Aug 02 09:33:11.875121 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:11.875083    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cs5vh"
Aug 02 09:33:11.875560 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:11.875209    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cs5vh" podUID="e6944ae2-aaed-4a4a-9c46-dda99772fa54"
Aug 02 09:33:12.875767 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:12.875734    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jmd"
Aug 02 09:33:12.875767 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:12.875754    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5mrn"
Aug 02 09:33:12.876301 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:12.875859    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jmd" podUID="3c47caf2-5d96-4343-ab9a-5da7195e209c"
Aug 02 09:33:12.876301 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:12.876001    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5mrn" podUID="2068f737-ee2f-4310-977e-27cb7e1d4f73"
Aug 02 09:33:13.876687 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:13.876650    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cs5vh"
Aug 02 09:33:13.877204 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:13.876781    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cs5vh" podUID="e6944ae2-aaed-4a4a-9c46-dda99772fa54"
Aug 02 09:33:14.445062 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:14.445028    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2068f737-ee2f-4310-977e-27cb7e1d4f73-metrics-certs\") pod \"network-metrics-daemon-v5mrn\" (UID: \"2068f737-ee2f-4310-977e-27cb7e1d4f73\") " pod="openshift-multus/network-metrics-daemon-v5mrn"
Aug 02 09:33:14.445219 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:14.445157    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 02 09:33:14.445219 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:14.445212    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2068f737-ee2f-4310-977e-27cb7e1d4f73-metrics-certs podName:2068f737-ee2f-4310-977e-27cb7e1d4f73 nodeName:}" failed. No retries permitted until 2026-08-02 09:33:30.445197557 +0000 UTC m=+34.160002982 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2068f737-ee2f-4310-977e-27cb7e1d4f73-metrics-certs") pod "network-metrics-daemon-v5mrn" (UID: "2068f737-ee2f-4310-977e-27cb7e1d4f73") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 02 09:33:14.545933 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:14.545899    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e6944ae2-aaed-4a4a-9c46-dda99772fa54-original-pull-secret\") pod \"global-pull-secret-syncer-cs5vh\" (UID: \"e6944ae2-aaed-4a4a-9c46-dda99772fa54\") " pod="kube-system/global-pull-secret-syncer-cs5vh"
Aug 02 09:33:14.546113 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:14.546064    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 02 09:33:14.546228 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:14.546135    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6944ae2-aaed-4a4a-9c46-dda99772fa54-original-pull-secret podName:e6944ae2-aaed-4a4a-9c46-dda99772fa54 nodeName:}" failed. No retries permitted until 2026-08-02 09:33:30.546117071 +0000 UTC m=+34.260922498 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e6944ae2-aaed-4a4a-9c46-dda99772fa54-original-pull-secret") pod "global-pull-secret-syncer-cs5vh" (UID: "e6944ae2-aaed-4a4a-9c46-dda99772fa54") : object "kube-system"/"original-pull-secret" not registered
Aug 02 09:33:14.647019 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:14.646987    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtx9x\" (UniqueName: \"kubernetes.io/projected/3c47caf2-5d96-4343-ab9a-5da7195e209c-kube-api-access-mtx9x\") pod \"network-check-target-m9jmd\" (UID: \"3c47caf2-5d96-4343-ab9a-5da7195e209c\") " pod="openshift-network-diagnostics/network-check-target-m9jmd"
Aug 02 09:33:14.647205 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:14.647170    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 02 09:33:14.647205 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:14.647202    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 02 09:33:14.647331 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:14.647215    2564 projected.go:196] Error preparing data for projected volume kube-api-access-mtx9x for pod openshift-network-diagnostics/network-check-target-m9jmd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 02 09:33:14.647331 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:14.647296    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c47caf2-5d96-4343-ab9a-5da7195e209c-kube-api-access-mtx9x podName:3c47caf2-5d96-4343-ab9a-5da7195e209c nodeName:}" failed. No retries permitted until 2026-08-02 09:33:30.647277082 +0000 UTC m=+34.362082521 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mtx9x" (UniqueName: "kubernetes.io/projected/3c47caf2-5d96-4343-ab9a-5da7195e209c-kube-api-access-mtx9x") pod "network-check-target-m9jmd" (UID: "3c47caf2-5d96-4343-ab9a-5da7195e209c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 02 09:33:14.875384 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:14.875304    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jmd"
Aug 02 09:33:14.875384 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:14.875321    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5mrn"
Aug 02 09:33:14.875600 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:14.875494    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jmd" podUID="3c47caf2-5d96-4343-ab9a-5da7195e209c"
Aug 02 09:33:14.875600 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:14.875481    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5mrn" podUID="2068f737-ee2f-4310-977e-27cb7e1d4f73"
Aug 02 09:33:15.875659 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:15.875622    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cs5vh"
Aug 02 09:33:15.876091 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:15.875785    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cs5vh" podUID="e6944ae2-aaed-4a4a-9c46-dda99772fa54"
Aug 02 09:33:16.878997 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:16.876439    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5mrn"
Aug 02 09:33:16.878997 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:16.876481    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jmd"
Aug 02 09:33:16.878997 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:16.876660    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5mrn" podUID="2068f737-ee2f-4310-977e-27cb7e1d4f73"
Aug 02 09:33:16.878997 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:16.876772    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jmd" podUID="3c47caf2-5d96-4343-ab9a-5da7195e209c"
Aug 02 09:33:17.875629 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:17.875595    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cs5vh"
Aug 02 09:33:17.875806 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:17.875719    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cs5vh" podUID="e6944ae2-aaed-4a4a-9c46-dda99772fa54"
Aug 02 09:33:18.875000 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:18.874974    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5mrn"
Aug 02 09:33:18.875393 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:18.875099    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5mrn" podUID="2068f737-ee2f-4310-977e-27cb7e1d4f73"
Aug 02 09:33:18.875393 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:18.875157    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jmd"
Aug 02 09:33:18.875393 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:18.875282    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jmd" podUID="3c47caf2-5d96-4343-ab9a-5da7195e209c"
Aug 02 09:33:19.068177 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:19.067904    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fhzz9" event={"ID":"3f97d5c9-1a25-43c5-8050-9c619be890e4","Type":"ContainerStarted","Data":"f22df0252fcf5f4a2304bf7ec0b84579df6e79cde8dcb79305bf2ead8862ecef"}
Aug 02 09:33:19.074612 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:19.073714    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fj9zj" event={"ID":"95220006-a44c-46f9-b380-f771acbd1eac","Type":"ContainerStarted","Data":"7e7ba56e5e3be70aea3295bfb1f59a265090a4698475e2320fc4ef261f1186ae"}
Aug 02 09:33:19.082107 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:19.082067    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-fhzz9" podStartSLOduration=6.941511293 podStartE2EDuration="22.082053183s" totalImagesPullingTime="15.14054189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 09:32:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:33:19.08146898 +0000 UTC m=+22.796274425" watchObservedRunningTime="2026-08-02 09:33:19.082053183 +0000 UTC m=+22.796858631"
Aug 02 09:33:19.101861 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:19.101375    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fj9zj" podStartSLOduration=3.249679745 podStartE2EDuration="22.101353256s" totalImagesPullingTime="18.851673511s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-02 09:32:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:33:19.10120221 +0000 UTC m=+22.816007658" watchObservedRunningTime="2026-08-02 09:33:19.101353256 +0000 UTC m=+22.816158706"
Aug 02 09:33:19.851177 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:19.850952    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:19.861907 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:19.861888    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-fhzz9"
Aug 02 09:33:19.875098 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:19.875078    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cs5vh"
Aug 02 09:33:19.875668 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:19.875173    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cs5vh" podUID="e6944ae2-aaed-4a4a-9c46-dda99772fa54"
Aug 02 09:33:19.946356 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:19.946321    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:20.008782 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:20.008748    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:20.066952 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:20.066854    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:20.077849 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:20.077823    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-59x9w" event={"ID":"90afa8c6-44be-4641-aa6b-90b51f5c1c6b","Type":"ContainerStarted","Data":"d7b195804278ff1eb566bc6a4d4f720f0aa33fdaf655ccce81874583d44e7f3e"}
Aug 02 09:33:20.077958 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:20.077912    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-59x9w" event={"ID":"90afa8c6-44be-4641-aa6b-90b51f5c1c6b","Type":"ContainerStarted","Data":"5be8a2292bd0a806f69957d04492a6819db80a887a987483c6d904bf294501ff"}
Aug 02 09:33:20.078013 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:20.077964    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-59x9w" event={"ID":"90afa8c6-44be-4641-aa6b-90b51f5c1c6b","Type":"ContainerStarted","Data":"c0fb66f32fdb1024737113cf77d40a8e78385924aefb5a98efcdf81e7d031fae"}
Aug 02 09:33:20.078049 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:20.078011    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-59x9w" event={"ID":"90afa8c6-44be-4641-aa6b-90b51f5c1c6b","Type":"ContainerStarted","Data":"6e01ac67b7775722b245c825d948e2e0858bb67c6b3609baa3989637a18c918f"}
Aug 02 09:33:20.078091 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:20.078054    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-59x9w" event={"ID":"90afa8c6-44be-4641-aa6b-90b51f5c1c6b","Type":"ContainerStarted","Data":"3cf2b38189341b8a28beb9561db8a55f4939a168ded0b433b7d727ce9179a689"}
Aug 02 09:33:20.078138 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:20.078089    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-59x9w" event={"ID":"90afa8c6-44be-4641-aa6b-90b51f5c1c6b","Type":"ContainerStarted","Data":"f859bd03806c090cc69f8ad6264cf640f06957381c1e64a548b1cc53bfcd8ff9"}
Aug 02 09:33:20.079212 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:20.079194    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ht722" event={"ID":"9901b738-6f94-48a9-9282-a051ce9d37ae","Type":"ContainerStarted","Data":"2241a0f694f5f75c6e67cf0da30aacb4270de1581d7fcbe447548e89ab809492"}
Aug 02 09:33:20.080464 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:20.080443    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8lm5w" event={"ID":"6b0cc25e-0367-4ac6-b094-1caf4d2023ab","Type":"ContainerStarted","Data":"cfbc80dbe98ecb26108726d94167b8179c47c45c437bda761f7ad16b69725dbb"}
Aug 02 09:33:20.081754 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:20.081734    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4f6pd" event={"ID":"032156ef-c7a4-4cd5-a991-4c91f2a67b4b","Type":"ContainerStarted","Data":"be13b9d93d2d3a9e4e17326fb7952108681d42bec32c7826efd8636c055e8ced"}
Aug 02 09:33:20.083081 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:20.083062    2564 generic.go:356] "Generic (PLEG): container finished" podID="0fb6399d-1991-4b8e-920c-a1086603b643" containerID="71984cd4479b4fad8d4afba282ee37e3f6d36ca8bf7e63c86e8be8a4b8a3d9b9" exitCode=0
Aug 02 09:33:20.083172 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:20.083116    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qgz2q" event={"ID":"0fb6399d-1991-4b8e-920c-a1086603b643","Type":"ContainerDied","Data":"71984cd4479b4fad8d4afba282ee37e3f6d36ca8bf7e63c86e8be8a4b8a3d9b9"}
Aug 02 09:33:20.084535 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:20.084515    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfhxd" event={"ID":"6ddc3ad8-170f-49b5-8a1c-6fe966dd658a","Type":"ContainerStarted","Data":"b37506df3cb6d7d7edbbe68db3ef42217a8b775a15e8844aca6539f9b7020665"}
Aug 02 09:33:20.095425 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:20.095381    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-ht722" podStartSLOduration=4.200304248 podStartE2EDuration="23.095369859s" totalImagesPullingTime="18.895065611s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-02 09:32:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:33:20.094959516 +0000 UTC m=+23.809764963" watchObservedRunningTime="2026-08-02 09:33:20.095369859 +0000 UTC m=+23.810175307"
Aug 02 09:33:20.109613 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:20.109536    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8lm5w" podStartSLOduration=4.296002778 podStartE2EDuration="23.10952313s" totalImagesPullingTime="18.813520352s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-02 09:32:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:33:20.109019939 +0000 UTC m=+23.823825388" watchObservedRunningTime="2026-08-02 09:33:20.10952313 +0000 UTC m=+23.824328584"
Aug 02 09:33:20.121518 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:20.121481    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-4f6pd" podStartSLOduration=4.317220283 podStartE2EDuration="23.121467913s" totalImagesPullingTime="18.80424763s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-02 09:32:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:33:20.121081396 +0000 UTC m=+23.835886845" watchObservedRunningTime="2026-08-02 09:33:20.121467913 +0000 UTC m=+23.836273360"
Aug 02 09:33:20.258053 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:20.258024    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:20.361921 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:20.361842    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:20.516943 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:20.516919    2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Aug 02 09:33:20.604262 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:20.604205    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:20.664690 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:20.664610    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:20.771065 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:20.770927    2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-08-02T09:33:20.516937264Z","UUID":"6caf05cc-929d-4a49-a874-76922d124c7a","Handler":null,"Name":"","Endpoint":""}
Aug 02 09:33:20.772778 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:20.772757    2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Aug 02 09:33:20.772892 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:20.772798    2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Aug 02 09:33:20.875097 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:20.875070    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5mrn"
Aug 02 09:33:20.875097 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:20.875089    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jmd"
Aug 02 09:33:20.875665 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:20.875188    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5mrn" podUID="2068f737-ee2f-4310-977e-27cb7e1d4f73"
Aug 02 09:33:20.875665 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:20.875368    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jmd" podUID="3c47caf2-5d96-4343-ab9a-5da7195e209c"
Aug 02 09:33:21.001048 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:21.000978    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:21.089596 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:21.089563    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfhxd" event={"ID":"6ddc3ad8-170f-49b5-8a1c-6fe966dd658a","Type":"ContainerStarted","Data":"81d6f1957893f8f84dca73081567f5d81481f0ebaecaabb505bb209928688276"}
Aug 02 09:33:21.091571 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:21.091176    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-878zb" event={"ID":"2e5695b0-d7f7-4770-93a8-b9bcc79efad1","Type":"ContainerStarted","Data":"a972a866189d04f71746ef27d815b030ad71fa3829d0f1790542a68ca0f127f5"}
Aug 02 09:33:21.092723 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:21.092693    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:21.104119 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:21.104082    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-878zb" podStartSLOduration=5.298709226 podStartE2EDuration="24.104069375s" totalImagesPullingTime="18.805360149s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-02 09:32:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:33:21.103662325 +0000 UTC m=+24.818467773" watchObservedRunningTime="2026-08-02 09:33:21.104069375 +0000 UTC m=+24.818874822"
Aug 02 09:33:21.205660 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:21.205629    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:21.261672 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:21.261580    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:21.794789 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:21.794766    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:21.851797 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:21.851524    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:21.874994 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:21.874973    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cs5vh"
Aug 02 09:33:21.875140 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:21.875101    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cs5vh" podUID="e6944ae2-aaed-4a4a-9c46-dda99772fa54"
Aug 02 09:33:22.095593 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:22.095555    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfhxd" event={"ID":"6ddc3ad8-170f-49b5-8a1c-6fe966dd658a","Type":"ContainerStarted","Data":"74dce3a2b6087b6968f55c5ad88b73ae5e2f28b9e65ac16767c31350d64ca80f"}
Aug 02 09:33:22.099214 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:22.099184    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-59x9w" event={"ID":"90afa8c6-44be-4641-aa6b-90b51f5c1c6b","Type":"ContainerStarted","Data":"5b8bd243f4d2fa3c8ac1d6a0bedbdfd2957f5e5d31363b0f7814f0b8a48ecb85"}
Aug 02 09:33:22.110298 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:22.110194    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tfhxd" podStartSLOduration=3.734837065 podStartE2EDuration="25.110174764s" totalImagesPullingTime="21.375337699s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-02 09:32:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:33:22.10955411 +0000 UTC m=+25.824359559" watchObservedRunningTime="2026-08-02 09:33:22.110174764 +0000 UTC m=+25.824980216"
Aug 02 09:33:22.203903 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:22.203873    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:22.305044 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:22.305013    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:22.589915 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:22.589833    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-fhzz9"
Aug 02 09:33:22.590564 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:22.590533    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-fhzz9"
Aug 02 09:33:22.659511 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:22.659486    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:22.875919 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:22.875822    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jmd"
Aug 02 09:33:22.876495 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:22.875959    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jmd" podUID="3c47caf2-5d96-4343-ab9a-5da7195e209c"
Aug 02 09:33:22.876495 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:22.876000    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5mrn"
Aug 02 09:33:22.876495 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:22.876098    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5mrn" podUID="2068f737-ee2f-4310-977e-27cb7e1d4f73"
Aug 02 09:33:23.003575 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:23.003542    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:23.102247 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:23.102205    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-fhzz9"
Aug 02 09:33:23.158549 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:23.158478    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:23.401448 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:23.401414    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:23.454490 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:23.454417    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:23.859408 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:23.859343    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:23.875422 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:23.875381    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cs5vh"
Aug 02 09:33:23.875556 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:23.875493    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cs5vh" podUID="e6944ae2-aaed-4a4a-9c46-dda99772fa54"
Aug 02 09:33:23.959942 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:23.959908    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:24.197183 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:24.197118    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:24.298647 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:24.298580    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:24.598851 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:24.598665    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:24.651662 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:24.651639    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:24.874801 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:24.874736    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5mrn"
Aug 02 09:33:24.874923 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:24.874847    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5mrn" podUID="2068f737-ee2f-4310-977e-27cb7e1d4f73"
Aug 02 09:33:24.874923 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:24.874897    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jmd"
Aug 02 09:33:24.875007 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:24.874977    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jmd" podUID="3c47caf2-5d96-4343-ab9a-5da7195e209c"
Aug 02 09:33:25.109488 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:25.109456    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-59x9w" event={"ID":"90afa8c6-44be-4641-aa6b-90b51f5c1c6b","Type":"ContainerStarted","Data":"d4da78b83bfb91cb93d44314771d9a91605dc080c10e5623e0d7c762d32c3c97"}
Aug 02 09:33:25.110185 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:25.109690    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:33:25.111203 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:25.111177    2564 generic.go:356] "Generic (PLEG): container finished" podID="0fb6399d-1991-4b8e-920c-a1086603b643" containerID="5b4e83faa2a0ec0e9649624dd377819387b75d6bd1f2cf6777e5f82c46192c42" exitCode=0
Aug 02 09:33:25.111320 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:25.111212    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qgz2q" event={"ID":"0fb6399d-1991-4b8e-920c-a1086603b643","Type":"ContainerDied","Data":"5b4e83faa2a0ec0e9649624dd377819387b75d6bd1f2cf6777e5f82c46192c42"}
Aug 02 09:33:25.124543 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:25.124524    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:33:25.136020 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:25.135948    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-59x9w" podStartSLOduration=9.263612929 podStartE2EDuration="28.135932313s" totalImagesPullingTime="18.872319384s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-02 09:32:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:33:25.132663217 +0000 UTC m=+28.847468662" watchObservedRunningTime="2026-08-02 09:33:25.135932313 +0000 UTC m=+28.850737761"
Aug 02 09:33:25.875378 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:25.875346    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cs5vh"
Aug 02 09:33:25.875680 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:25.875527    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cs5vh" podUID="e6944ae2-aaed-4a4a-9c46-dda99772fa54"
Aug 02 09:33:26.115974 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:26.115752    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qgz2q" event={"ID":"0fb6399d-1991-4b8e-920c-a1086603b643","Type":"ContainerStarted","Data":"5ce1d613ee1bd3c620c27043ffa6b40bf6a48b34d7934c437a04f901b454d3a4"}
Aug 02 09:33:26.116301 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:26.116244    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:33:26.116344 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:26.116316    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:33:26.130493 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:26.130443    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:33:26.225146 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:26.225119    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-m9jmd"]
Aug 02 09:33:26.225316 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:26.225254    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jmd"
Aug 02 09:33:26.225381 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:26.225356    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jmd" podUID="3c47caf2-5d96-4343-ab9a-5da7195e209c"
Aug 02 09:33:26.227566 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:26.227542    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cs5vh"]
Aug 02 09:33:26.227702 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:26.227609    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cs5vh"
Aug 02 09:33:26.227702 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:26.227681    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cs5vh" podUID="e6944ae2-aaed-4a4a-9c46-dda99772fa54"
Aug 02 09:33:26.235777 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:26.235756    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-v5mrn"]
Aug 02 09:33:26.235896 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:26.235843    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5mrn"
Aug 02 09:33:26.235959 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:26.235928    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5mrn" podUID="2068f737-ee2f-4310-977e-27cb7e1d4f73"
Aug 02 09:33:27.119570 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:27.119539    2564 generic.go:356] "Generic (PLEG): container finished" podID="0fb6399d-1991-4b8e-920c-a1086603b643" containerID="5ce1d613ee1bd3c620c27043ffa6b40bf6a48b34d7934c437a04f901b454d3a4" exitCode=0
Aug 02 09:33:27.119911 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:27.119621    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qgz2q" event={"ID":"0fb6399d-1991-4b8e-920c-a1086603b643","Type":"ContainerDied","Data":"5ce1d613ee1bd3c620c27043ffa6b40bf6a48b34d7934c437a04f901b454d3a4"}
Aug 02 09:33:27.875331 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:27.875301    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5mrn"
Aug 02 09:33:27.875452 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:27.875308    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jmd"
Aug 02 09:33:27.875452 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:27.875404    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5mrn" podUID="2068f737-ee2f-4310-977e-27cb7e1d4f73"
Aug 02 09:33:27.875522 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:27.875308    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cs5vh"
Aug 02 09:33:27.875554 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:27.875457    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jmd" podUID="3c47caf2-5d96-4343-ab9a-5da7195e209c"
Aug 02 09:33:27.875587 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:27.875560    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cs5vh" podUID="e6944ae2-aaed-4a4a-9c46-dda99772fa54"
Aug 02 09:33:28.124488 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:28.124149    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qgz2q" event={"ID":"0fb6399d-1991-4b8e-920c-a1086603b643","Type":"ContainerStarted","Data":"ffab221aa50f8120ead03d83ca1e2e4deaba9bbfa4cdc57af56fc6449a2a1726"}
Aug 02 09:33:29.128484 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:29.128447    2564 generic.go:356] "Generic (PLEG): container finished" podID="0fb6399d-1991-4b8e-920c-a1086603b643" containerID="ffab221aa50f8120ead03d83ca1e2e4deaba9bbfa4cdc57af56fc6449a2a1726" exitCode=0
Aug 02 09:33:29.128982 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:29.128493    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qgz2q" event={"ID":"0fb6399d-1991-4b8e-920c-a1086603b643","Type":"ContainerDied","Data":"ffab221aa50f8120ead03d83ca1e2e4deaba9bbfa4cdc57af56fc6449a2a1726"}
Aug 02 09:33:29.397707 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:29.397634    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:29.495754 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:29.495723    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:29.596602 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:29.596551    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:29.791355 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:29.791248    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:29.845428 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:29.845396    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:29.875281 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:29.875251    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cs5vh"
Aug 02 09:33:29.875447 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:29.875251    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jmd"
Aug 02 09:33:29.875447 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:29.875370    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cs5vh" podUID="e6944ae2-aaed-4a4a-9c46-dda99772fa54"
Aug 02 09:33:29.875447 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:29.875251    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5mrn"
Aug 02 09:33:29.875447 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:29.875441    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9jmd" podUID="3c47caf2-5d96-4343-ab9a-5da7195e209c"
Aug 02 09:33:29.875616 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:29.875497    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v5mrn" podUID="2068f737-ee2f-4310-977e-27cb7e1d4f73"
Aug 02 09:33:30.062429 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:30.062383    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:30.155137 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:30.155092    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:30.381688 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:30.381609    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:30.452459 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:30.452431    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:30.466292 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:30.466269    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2068f737-ee2f-4310-977e-27cb7e1d4f73-metrics-certs\") pod \"network-metrics-daemon-v5mrn\" (UID: \"2068f737-ee2f-4310-977e-27cb7e1d4f73\") " pod="openshift-multus/network-metrics-daemon-v5mrn"
Aug 02 09:33:30.466427 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:30.466412    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 02 09:33:30.466483 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:30.466475    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2068f737-ee2f-4310-977e-27cb7e1d4f73-metrics-certs podName:2068f737-ee2f-4310-977e-27cb7e1d4f73 nodeName:}" failed. No retries permitted until 2026-08-02 09:34:02.466460725 +0000 UTC m=+66.181266163 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2068f737-ee2f-4310-977e-27cb7e1d4f73-metrics-certs") pod "network-metrics-daemon-v5mrn" (UID: "2068f737-ee2f-4310-977e-27cb7e1d4f73") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 02 09:33:30.503019 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:30.502987    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:30.566959 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:30.566934    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e6944ae2-aaed-4a4a-9c46-dda99772fa54-original-pull-secret\") pod \"global-pull-secret-syncer-cs5vh\" (UID: \"e6944ae2-aaed-4a4a-9c46-dda99772fa54\") " pod="kube-system/global-pull-secret-syncer-cs5vh"
Aug 02 09:33:30.567099 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:30.567075    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 02 09:33:30.567171 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:30.567126    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6944ae2-aaed-4a4a-9c46-dda99772fa54-original-pull-secret podName:e6944ae2-aaed-4a4a-9c46-dda99772fa54 nodeName:}" failed. No retries permitted until 2026-08-02 09:34:02.567112802 +0000 UTC m=+66.281918231 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e6944ae2-aaed-4a4a-9c46-dda99772fa54-original-pull-secret") pod "global-pull-secret-syncer-cs5vh" (UID: "e6944ae2-aaed-4a4a-9c46-dda99772fa54") : object "kube-system"/"original-pull-secret" not registered
Aug 02 09:33:30.667553 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:30.667471    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtx9x\" (UniqueName: \"kubernetes.io/projected/3c47caf2-5d96-4343-ab9a-5da7195e209c-kube-api-access-mtx9x\") pod \"network-check-target-m9jmd\" (UID: \"3c47caf2-5d96-4343-ab9a-5da7195e209c\") " pod="openshift-network-diagnostics/network-check-target-m9jmd"
Aug 02 09:33:30.667680 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:30.667636    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 02 09:33:30.667680 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:30.667661    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 02 09:33:30.667680 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:30.667671    2564 projected.go:196] Error preparing data for projected volume kube-api-access-mtx9x for pod openshift-network-diagnostics/network-check-target-m9jmd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 02 09:33:30.667774 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:30.667719    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c47caf2-5d96-4343-ab9a-5da7195e209c-kube-api-access-mtx9x podName:3c47caf2-5d96-4343-ab9a-5da7195e209c nodeName:}" failed. No retries permitted until 2026-08-02 09:34:02.667704172 +0000 UTC m=+66.382509614 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mtx9x" (UniqueName: "kubernetes.io/projected/3c47caf2-5d96-4343-ab9a-5da7195e209c-kube-api-access-mtx9x") pod "network-check-target-m9jmd" (UID: "3c47caf2-5d96-4343-ab9a-5da7195e209c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 02 09:33:30.800214 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:30.800175    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:30.879031 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:30.879002    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:30.947437 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:30.947362    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4f6pd_032156ef-c7a4-4cd5-a991-4c91f2a67b4b/dns-node-resolver/0.log"
Aug 02 09:33:31.559035 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.559006    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-232.ec2.internal" event="NodeReady"
Aug 02 09:33:31.559507 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.559153    2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Aug 02 09:33:31.589497 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.589465    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5f6b7788d4-hl9g6"]
Aug 02 09:33:31.593484 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.593463    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f6b7788d4-hl9g6"
Aug 02 09:33:31.595538 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.595511    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Aug 02 09:33:31.595651 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.595635    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Aug 02 09:33:31.595727 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.595665    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-f75jg\""
Aug 02 09:33:31.595789 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.595748    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Aug 02 09:33:31.599617 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.599595    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-pvvv7"]
Aug 02 09:33:31.602862 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.602199    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Aug 02 09:33:31.605581 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.605498    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pvvv7"
Aug 02 09:33:31.606458 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.606437    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-89bsp"]
Aug 02 09:33:31.607601 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.607416    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Aug 02 09:33:31.607601 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.607556    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Aug 02 09:33:31.607767 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.607601    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Aug 02 09:33:31.607767 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.607646    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4s9vb\""
Aug 02 09:33:31.610851 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.610804    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f6b7788d4-hl9g6"]
Aug 02 09:33:31.611029 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.610910    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pvvv7"]
Aug 02 09:33:31.611029 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.610909    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-89bsp"
Aug 02 09:33:31.612996 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.612971    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Aug 02 09:33:31.613263 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.613224    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-49fv9\""
Aug 02 09:33:31.613391 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.613370    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Aug 02 09:33:31.613503 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.613433    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Aug 02 09:33:31.613651 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.613633    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Aug 02 09:33:31.618773 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.618753    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-89bsp"]
Aug 02 09:33:31.619225 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.619207    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7f8lc"]
Aug 02 09:33:31.623110 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.623092    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7f8lc"
Aug 02 09:33:31.624904 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.624888    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Aug 02 09:33:31.624971 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.624896    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Aug 02 09:33:31.624971 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.624956    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zv4ml\""
Aug 02 09:33:31.628983 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.628845    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7f8lc"]
Aug 02 09:33:31.674580 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.674491    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/88085931-e03b-4e83-a890-e9b7dfafeda1-ca-trust-extracted\") pod \"image-registry-5f6b7788d4-hl9g6\" (UID: \"88085931-e03b-4e83-a890-e9b7dfafeda1\") " pod="openshift-image-registry/image-registry-5f6b7788d4-hl9g6"
Aug 02 09:33:31.674580 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.674562    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/88085931-e03b-4e83-a890-e9b7dfafeda1-registry-certificates\") pod \"image-registry-5f6b7788d4-hl9g6\" (UID: \"88085931-e03b-4e83-a890-e9b7dfafeda1\") " pod="openshift-image-registry/image-registry-5f6b7788d4-hl9g6"
Aug 02 09:33:31.674806 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.674631    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/88085931-e03b-4e83-a890-e9b7dfafeda1-installation-pull-secrets\") pod \"image-registry-5f6b7788d4-hl9g6\" (UID: \"88085931-e03b-4e83-a890-e9b7dfafeda1\") " pod="openshift-image-registry/image-registry-5f6b7788d4-hl9g6"
Aug 02 09:33:31.674806 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.674707    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/88085931-e03b-4e83-a890-e9b7dfafeda1-trusted-ca\") pod \"image-registry-5f6b7788d4-hl9g6\" (UID: \"88085931-e03b-4e83-a890-e9b7dfafeda1\") " pod="openshift-image-registry/image-registry-5f6b7788d4-hl9g6"
Aug 02 09:33:31.674806 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.674738    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/88085931-e03b-4e83-a890-e9b7dfafeda1-bound-sa-token\") pod \"image-registry-5f6b7788d4-hl9g6\" (UID: \"88085931-e03b-4e83-a890-e9b7dfafeda1\") " pod="openshift-image-registry/image-registry-5f6b7788d4-hl9g6"
Aug 02 09:33:31.674806 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.674765    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc75s\" (UniqueName: \"kubernetes.io/projected/88085931-e03b-4e83-a890-e9b7dfafeda1-kube-api-access-kc75s\") pod \"image-registry-5f6b7788d4-hl9g6\" (UID: \"88085931-e03b-4e83-a890-e9b7dfafeda1\") " pod="openshift-image-registry/image-registry-5f6b7788d4-hl9g6"
Aug 02 09:33:31.674806 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.674799    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/88085931-e03b-4e83-a890-e9b7dfafeda1-image-registry-private-configuration\") pod \"image-registry-5f6b7788d4-hl9g6\" (UID: \"88085931-e03b-4e83-a890-e9b7dfafeda1\") " pod="openshift-image-registry/image-registry-5f6b7788d4-hl9g6"
Aug 02 09:33:31.674996 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.674824    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/88085931-e03b-4e83-a890-e9b7dfafeda1-registry-tls\") pod \"image-registry-5f6b7788d4-hl9g6\" (UID: \"88085931-e03b-4e83-a890-e9b7dfafeda1\") " pod="openshift-image-registry/image-registry-5f6b7788d4-hl9g6"
Aug 02 09:33:31.726900 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.726865    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:31.775587 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.775550    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/88085931-e03b-4e83-a890-e9b7dfafeda1-registry-certificates\") pod \"image-registry-5f6b7788d4-hl9g6\" (UID: \"88085931-e03b-4e83-a890-e9b7dfafeda1\") " pod="openshift-image-registry/image-registry-5f6b7788d4-hl9g6"
Aug 02 09:33:31.775587 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.775588    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/88085931-e03b-4e83-a890-e9b7dfafeda1-installation-pull-secrets\") pod \"image-registry-5f6b7788d4-hl9g6\" (UID: \"88085931-e03b-4e83-a890-e9b7dfafeda1\") " pod="openshift-image-registry/image-registry-5f6b7788d4-hl9g6"
Aug 02 09:33:31.775806 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.775616    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/80c7ce46-ead1-48ad-9c47-296c0124131d-cert\") pod \"ingress-canary-pvvv7\" (UID: \"80c7ce46-ead1-48ad-9c47-296c0124131d\") " pod="openshift-ingress-canary/ingress-canary-pvvv7"
Aug 02 09:33:31.775806 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.775642    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b6e3c57-2d9d-419a-a7e2-c419d29faecc-tmp-dir\") pod \"dns-default-7f8lc\" (UID: \"6b6e3c57-2d9d-419a-a7e2-c419d29faecc\") " pod="openshift-dns/dns-default-7f8lc"
Aug 02 09:33:31.775918 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.775863    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b6e3c57-2d9d-419a-a7e2-c419d29faecc-metrics-tls\") pod \"dns-default-7f8lc\" (UID: \"6b6e3c57-2d9d-419a-a7e2-c419d29faecc\") " pod="openshift-dns/dns-default-7f8lc"
Aug 02 09:33:31.775973 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.775915    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/88085931-e03b-4e83-a890-e9b7dfafeda1-trusted-ca\") pod \"image-registry-5f6b7788d4-hl9g6\" (UID: \"88085931-e03b-4e83-a890-e9b7dfafeda1\") " pod="openshift-image-registry/image-registry-5f6b7788d4-hl9g6"
Aug 02 09:33:31.775973 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.775948    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q5p7\" (UniqueName: \"kubernetes.io/projected/80c7ce46-ead1-48ad-9c47-296c0124131d-kube-api-access-2q5p7\") pod \"ingress-canary-pvvv7\" (UID: \"80c7ce46-ead1-48ad-9c47-296c0124131d\") " pod="openshift-ingress-canary/ingress-canary-pvvv7"
Aug 02 09:33:31.776071 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.775973    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tp7r\" (UniqueName: \"kubernetes.io/projected/6b6e3c57-2d9d-419a-a7e2-c419d29faecc-kube-api-access-9tp7r\") pod \"dns-default-7f8lc\" (UID: \"6b6e3c57-2d9d-419a-a7e2-c419d29faecc\") " pod="openshift-dns/dns-default-7f8lc"
Aug 02 09:33:31.776071 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.776024    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lrs5\" (UniqueName: \"kubernetes.io/projected/0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb-kube-api-access-9lrs5\") pod \"insights-runtime-extractor-89bsp\" (UID: \"0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb\") " pod="openshift-insights/insights-runtime-extractor-89bsp"
Aug 02 09:33:31.776071 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.776051    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb-crio-socket\") pod \"insights-runtime-extractor-89bsp\" (UID: \"0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb\") " pod="openshift-insights/insights-runtime-extractor-89bsp"
Aug 02 09:33:31.776215 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.776079    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/88085931-e03b-4e83-a890-e9b7dfafeda1-bound-sa-token\") pod \"image-registry-5f6b7788d4-hl9g6\" (UID: \"88085931-e03b-4e83-a890-e9b7dfafeda1\") " pod="openshift-image-registry/image-registry-5f6b7788d4-hl9g6"
Aug 02 09:33:31.776215 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.776104    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-89bsp\" (UID: \"0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb\") " pod="openshift-insights/insights-runtime-extractor-89bsp"
Aug 02 09:33:31.776215 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.776138    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc75s\" (UniqueName: \"kubernetes.io/projected/88085931-e03b-4e83-a890-e9b7dfafeda1-kube-api-access-kc75s\") pod \"image-registry-5f6b7788d4-hl9g6\" (UID: \"88085931-e03b-4e83-a890-e9b7dfafeda1\") " pod="openshift-image-registry/image-registry-5f6b7788d4-hl9g6"
Aug 02 09:33:31.776215 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.776159    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6b6e3c57-2d9d-419a-a7e2-c419d29faecc-config-volume\") pod \"dns-default-7f8lc\" (UID: \"6b6e3c57-2d9d-419a-a7e2-c419d29faecc\") " pod="openshift-dns/dns-default-7f8lc"
Aug 02 09:33:31.776215 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.776174    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-89bsp\" (UID: \"0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb\") " pod="openshift-insights/insights-runtime-extractor-89bsp"
Aug 02 09:33:31.776215 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.776199    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/88085931-e03b-4e83-a890-e9b7dfafeda1-image-registry-private-configuration\") pod \"image-registry-5f6b7788d4-hl9g6\" (UID: \"88085931-e03b-4e83-a890-e9b7dfafeda1\") " pod="openshift-image-registry/image-registry-5f6b7788d4-hl9g6"
Aug 02 09:33:31.776518 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.776214    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/88085931-e03b-4e83-a890-e9b7dfafeda1-registry-certificates\") pod \"image-registry-5f6b7788d4-hl9g6\" (UID: \"88085931-e03b-4e83-a890-e9b7dfafeda1\") " pod="openshift-image-registry/image-registry-5f6b7788d4-hl9g6"
Aug 02 09:33:31.776518 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.776225    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/88085931-e03b-4e83-a890-e9b7dfafeda1-registry-tls\") pod \"image-registry-5f6b7788d4-hl9g6\" (UID: \"88085931-e03b-4e83-a890-e9b7dfafeda1\") " pod="openshift-image-registry/image-registry-5f6b7788d4-hl9g6"
Aug 02 09:33:31.776518 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.776325    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb-data-volume\") pod \"insights-runtime-extractor-89bsp\" (UID: \"0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb\") " pod="openshift-insights/insights-runtime-extractor-89bsp"
Aug 02 09:33:31.776518 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.776354    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/88085931-e03b-4e83-a890-e9b7dfafeda1-ca-trust-extracted\") pod \"image-registry-5f6b7788d4-hl9g6\" (UID: \"88085931-e03b-4e83-a890-e9b7dfafeda1\") " pod="openshift-image-registry/image-registry-5f6b7788d4-hl9g6"
Aug 02 09:33:31.776702 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:31.776613    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 02 09:33:31.776702 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:31.776628    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f6b7788d4-hl9g6: secret "image-registry-tls" not found
Aug 02 09:33:31.776702 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:31.776692    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88085931-e03b-4e83-a890-e9b7dfafeda1-registry-tls podName:88085931-e03b-4e83-a890-e9b7dfafeda1 nodeName:}" failed. No retries permitted until 2026-08-02 09:33:32.276672067 +0000 UTC m=+35.991477512 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/88085931-e03b-4e83-a890-e9b7dfafeda1-registry-tls") pod "image-registry-5f6b7788d4-hl9g6" (UID: "88085931-e03b-4e83-a890-e9b7dfafeda1") : secret "image-registry-tls" not found
Aug 02 09:33:31.776702 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.776697    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/88085931-e03b-4e83-a890-e9b7dfafeda1-ca-trust-extracted\") pod \"image-registry-5f6b7788d4-hl9g6\" (UID: \"88085931-e03b-4e83-a890-e9b7dfafeda1\") " pod="openshift-image-registry/image-registry-5f6b7788d4-hl9g6"
Aug 02 09:33:31.776880 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.776741    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/88085931-e03b-4e83-a890-e9b7dfafeda1-trusted-ca\") pod \"image-registry-5f6b7788d4-hl9g6\" (UID: \"88085931-e03b-4e83-a890-e9b7dfafeda1\") " pod="openshift-image-registry/image-registry-5f6b7788d4-hl9g6"
Aug 02 09:33:31.780329 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.780183    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/88085931-e03b-4e83-a890-e9b7dfafeda1-image-registry-private-configuration\") pod \"image-registry-5f6b7788d4-hl9g6\" (UID: \"88085931-e03b-4e83-a890-e9b7dfafeda1\") " pod="openshift-image-registry/image-registry-5f6b7788d4-hl9g6"
Aug 02 09:33:31.780438 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.780188    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/88085931-e03b-4e83-a890-e9b7dfafeda1-installation-pull-secrets\") pod \"image-registry-5f6b7788d4-hl9g6\" (UID: \"88085931-e03b-4e83-a890-e9b7dfafeda1\") " pod="openshift-image-registry/image-registry-5f6b7788d4-hl9g6"
Aug 02 09:33:31.784155 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.784132    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/88085931-e03b-4e83-a890-e9b7dfafeda1-bound-sa-token\") pod \"image-registry-5f6b7788d4-hl9g6\" (UID: \"88085931-e03b-4e83-a890-e9b7dfafeda1\") " pod="openshift-image-registry/image-registry-5f6b7788d4-hl9g6"
Aug 02 09:33:31.784264 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.784252    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc75s\" (UniqueName: \"kubernetes.io/projected/88085931-e03b-4e83-a890-e9b7dfafeda1-kube-api-access-kc75s\") pod \"image-registry-5f6b7788d4-hl9g6\" (UID: \"88085931-e03b-4e83-a890-e9b7dfafeda1\") " pod="openshift-image-registry/image-registry-5f6b7788d4-hl9g6"
Aug 02 09:33:31.827792 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.827764    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:31.875810 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.875782    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jmd"
Aug 02 09:33:31.875937 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.875782    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5mrn"
Aug 02 09:33:31.876068 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.875791    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cs5vh"
Aug 02 09:33:31.876620 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.876599    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q5p7\" (UniqueName: \"kubernetes.io/projected/80c7ce46-ead1-48ad-9c47-296c0124131d-kube-api-access-2q5p7\") pod \"ingress-canary-pvvv7\" (UID: \"80c7ce46-ead1-48ad-9c47-296c0124131d\") " pod="openshift-ingress-canary/ingress-canary-pvvv7"
Aug 02 09:33:31.876719 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.876628    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tp7r\" (UniqueName: \"kubernetes.io/projected/6b6e3c57-2d9d-419a-a7e2-c419d29faecc-kube-api-access-9tp7r\") pod \"dns-default-7f8lc\" (UID: \"6b6e3c57-2d9d-419a-a7e2-c419d29faecc\") " pod="openshift-dns/dns-default-7f8lc"
Aug 02 09:33:31.876719 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.876650    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lrs5\" (UniqueName: \"kubernetes.io/projected/0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb-kube-api-access-9lrs5\") pod \"insights-runtime-extractor-89bsp\" (UID: \"0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb\") " pod="openshift-insights/insights-runtime-extractor-89bsp"
Aug 02 09:33:31.876719 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.876678    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb-crio-socket\") pod \"insights-runtime-extractor-89bsp\" (UID: \"0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb\") " pod="openshift-insights/insights-runtime-extractor-89bsp"
Aug 02 09:33:31.876719 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.876705    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-89bsp\" (UID: \"0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb\") " pod="openshift-insights/insights-runtime-extractor-89bsp"
Aug 02 09:33:31.876889 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.876734    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6b6e3c57-2d9d-419a-a7e2-c419d29faecc-config-volume\") pod \"dns-default-7f8lc\" (UID: \"6b6e3c57-2d9d-419a-a7e2-c419d29faecc\") " pod="openshift-dns/dns-default-7f8lc"
Aug 02 09:33:31.876889 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.876757    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-89bsp\" (UID: \"0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb\") " pod="openshift-insights/insights-runtime-extractor-89bsp"
Aug 02 09:33:31.876889 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.876813    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb-data-volume\") pod \"insights-runtime-extractor-89bsp\" (UID: \"0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb\") " pod="openshift-insights/insights-runtime-extractor-89bsp"
Aug 02 09:33:31.876889 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.876878    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/80c7ce46-ead1-48ad-9c47-296c0124131d-cert\") pod \"ingress-canary-pvvv7\" (UID: \"80c7ce46-ead1-48ad-9c47-296c0124131d\") " pod="openshift-ingress-canary/ingress-canary-pvvv7"
Aug 02 09:33:31.877064 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.876899    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b6e3c57-2d9d-419a-a7e2-c419d29faecc-tmp-dir\") pod \"dns-default-7f8lc\" (UID: \"6b6e3c57-2d9d-419a-a7e2-c419d29faecc\") " pod="openshift-dns/dns-default-7f8lc"
Aug 02 09:33:31.877064 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.876942    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b6e3c57-2d9d-419a-a7e2-c419d29faecc-metrics-tls\") pod \"dns-default-7f8lc\" (UID: \"6b6e3c57-2d9d-419a-a7e2-c419d29faecc\") " pod="openshift-dns/dns-default-7f8lc"
Aug 02 09:33:31.877064 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.877012    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb-crio-socket\") pod \"insights-runtime-extractor-89bsp\" (UID: \"0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb\") " pod="openshift-insights/insights-runtime-extractor-89bsp"
Aug 02 09:33:31.877064 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:31.877042    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 02 09:33:31.877205 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:31.877096    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b6e3c57-2d9d-419a-a7e2-c419d29faecc-metrics-tls podName:6b6e3c57-2d9d-419a-a7e2-c419d29faecc nodeName:}" failed. No retries permitted until 2026-08-02 09:33:32.377081242 +0000 UTC m=+36.091886688 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6b6e3c57-2d9d-419a-a7e2-c419d29faecc-metrics-tls") pod "dns-default-7f8lc" (UID: "6b6e3c57-2d9d-419a-a7e2-c419d29faecc") : secret "dns-default-metrics-tls" not found
Aug 02 09:33:31.877205 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:31.877160    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 02 09:33:31.877205 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:31.877194    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb-insights-runtime-extractor-tls podName:0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb nodeName:}" failed. No retries permitted until 2026-08-02 09:33:32.377182656 +0000 UTC m=+36.091988083 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-89bsp" (UID: "0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb") : secret "insights-runtime-extractor-tls" not found
Aug 02 09:33:31.877562 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.877548    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b6e3c57-2d9d-419a-a7e2-c419d29faecc-tmp-dir\") pod \"dns-default-7f8lc\" (UID: \"6b6e3c57-2d9d-419a-a7e2-c419d29faecc\") " pod="openshift-dns/dns-default-7f8lc"
Aug 02 09:33:31.877686 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.877644    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb-data-volume\") pod \"insights-runtime-extractor-89bsp\" (UID: \"0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb\") " pod="openshift-insights/insights-runtime-extractor-89bsp"
Aug 02 09:33:31.877785 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:31.877689    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 02 09:33:31.877785 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:31.877745    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb-kube-rbac-proxy-cm podName:0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb nodeName:}" failed. No retries permitted until 2026-08-02 09:33:32.377727875 +0000 UTC m=+36.092533322 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-89bsp" (UID: "0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb") : configmap "kube-rbac-proxy" not found
Aug 02 09:33:31.877904 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.877852    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6b6e3c57-2d9d-419a-a7e2-c419d29faecc-config-volume\") pod \"dns-default-7f8lc\" (UID: \"6b6e3c57-2d9d-419a-a7e2-c419d29faecc\") " pod="openshift-dns/dns-default-7f8lc"
Aug 02 09:33:31.878379 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.878182    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Aug 02 09:33:31.878379 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.878214    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Aug 02 09:33:31.878379 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.878281    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xt8nz\""
Aug 02 09:33:31.878379 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.878283    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Aug 02 09:33:31.878658 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.878412    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mbl9n\""
Aug 02 09:33:31.878658 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.878510    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Aug 02 09:33:31.879830 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.879809    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/80c7ce46-ead1-48ad-9c47-296c0124131d-cert\") pod \"ingress-canary-pvvv7\" (UID: \"80c7ce46-ead1-48ad-9c47-296c0124131d\") " pod="openshift-ingress-canary/ingress-canary-pvvv7"
Aug 02 09:33:31.884108 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.883955    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tp7r\" (UniqueName: \"kubernetes.io/projected/6b6e3c57-2d9d-419a-a7e2-c419d29faecc-kube-api-access-9tp7r\") pod \"dns-default-7f8lc\" (UID: \"6b6e3c57-2d9d-419a-a7e2-c419d29faecc\") " pod="openshift-dns/dns-default-7f8lc"
Aug 02 09:33:31.884295 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.884276    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q5p7\" (UniqueName: \"kubernetes.io/projected/80c7ce46-ead1-48ad-9c47-296c0124131d-kube-api-access-2q5p7\") pod \"ingress-canary-pvvv7\" (UID: \"80c7ce46-ead1-48ad-9c47-296c0124131d\") " pod="openshift-ingress-canary/ingress-canary-pvvv7"
Aug 02 09:33:31.886962 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.886944    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lrs5\" (UniqueName: \"kubernetes.io/projected/0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb-kube-api-access-9lrs5\") pod \"insights-runtime-extractor-89bsp\" (UID: \"0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb\") " pod="openshift-insights/insights-runtime-extractor-89bsp"
Aug 02 09:33:31.918326 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.918305    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pvvv7"
Aug 02 09:33:31.947625 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:31.947551    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8lm5w_6b0cc25e-0367-4ac6-b094-1caf4d2023ab/node-ca/0.log"
Aug 02 09:33:32.280844 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:32.280767    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/88085931-e03b-4e83-a890-e9b7dfafeda1-registry-tls\") pod \"image-registry-5f6b7788d4-hl9g6\" (UID: \"88085931-e03b-4e83-a890-e9b7dfafeda1\") " pod="openshift-image-registry/image-registry-5f6b7788d4-hl9g6"
Aug 02 09:33:32.281008 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:32.280931    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 02 09:33:32.281008 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:32.280950    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f6b7788d4-hl9g6: secret "image-registry-tls" not found
Aug 02 09:33:32.281114 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:32.281015    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88085931-e03b-4e83-a890-e9b7dfafeda1-registry-tls podName:88085931-e03b-4e83-a890-e9b7dfafeda1 nodeName:}" failed. No retries permitted until 2026-08-02 09:33:33.280993503 +0000 UTC m=+36.995798950 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/88085931-e03b-4e83-a890-e9b7dfafeda1-registry-tls") pod "image-registry-5f6b7788d4-hl9g6" (UID: "88085931-e03b-4e83-a890-e9b7dfafeda1") : secret "image-registry-tls" not found
Aug 02 09:33:32.371732 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:32.371702    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-qvjx6"]
Aug 02 09:33:32.375041 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:32.375022    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-98c88fbb5-qvjx6"
Aug 02 09:33:32.378942 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:32.378917    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Aug 02 09:33:32.378942 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:32.378931    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Aug 02 09:33:32.379117 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:32.378930    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Aug 02 09:33:32.379117 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:32.379084    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Aug 02 09:33:32.379221 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:32.379121    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-wskln\""
Aug 02 09:33:32.381414 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:32.381383    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b6e3c57-2d9d-419a-a7e2-c419d29faecc-metrics-tls\") pod \"dns-default-7f8lc\" (UID: \"6b6e3c57-2d9d-419a-a7e2-c419d29faecc\") " pod="openshift-dns/dns-default-7f8lc"
Aug 02 09:33:32.381514 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:32.381432    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-89bsp\" (UID: \"0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb\") " pod="openshift-insights/insights-runtime-extractor-89bsp"
Aug 02 09:33:32.381580 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:32.381536    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 02 09:33:32.381580 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:32.381541    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 02 09:33:32.381680 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:32.381594    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb-insights-runtime-extractor-tls podName:0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb nodeName:}" failed. No retries permitted until 2026-08-02 09:33:33.381578427 +0000 UTC m=+37.096383856 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-89bsp" (UID: "0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb") : secret "insights-runtime-extractor-tls" not found
Aug 02 09:33:32.381680 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:32.381613    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b6e3c57-2d9d-419a-a7e2-c419d29faecc-metrics-tls podName:6b6e3c57-2d9d-419a-a7e2-c419d29faecc nodeName:}" failed. No retries permitted until 2026-08-02 09:33:33.381602494 +0000 UTC m=+37.096407923 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6b6e3c57-2d9d-419a-a7e2-c419d29faecc-metrics-tls") pod "dns-default-7f8lc" (UID: "6b6e3c57-2d9d-419a-a7e2-c419d29faecc") : secret "dns-default-metrics-tls" not found
Aug 02 09:33:32.381680 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:32.381632    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-89bsp\" (UID: \"0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb\") " pod="openshift-insights/insights-runtime-extractor-89bsp"
Aug 02 09:33:32.381845 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:32.381696    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 02 09:33:32.381845 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:32.381734    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb-kube-rbac-proxy-cm podName:0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb nodeName:}" failed. No retries permitted until 2026-08-02 09:33:33.381723061 +0000 UTC m=+37.096528492 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-89bsp" (UID: "0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb") : configmap "kube-rbac-proxy" not found
Aug 02 09:33:32.386869 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:32.385892    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-qvjx6"]
Aug 02 09:33:32.409578 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:32.409554    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:32.460226 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:32.460195    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:32.482424 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:32.482400    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/52f9c956-153d-4f6f-a69d-3366a4e774de-signing-cabundle\") pod \"service-ca-98c88fbb5-qvjx6\" (UID: \"52f9c956-153d-4f6f-a69d-3366a4e774de\") " pod="openshift-service-ca/service-ca-98c88fbb5-qvjx6"
Aug 02 09:33:32.482527 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:32.482465    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/52f9c956-153d-4f6f-a69d-3366a4e774de-signing-key\") pod \"service-ca-98c88fbb5-qvjx6\" (UID: \"52f9c956-153d-4f6f-a69d-3366a4e774de\") " pod="openshift-service-ca/service-ca-98c88fbb5-qvjx6"
Aug 02 09:33:32.482582 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:32.482531    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59bhc\" (UniqueName: \"kubernetes.io/projected/52f9c956-153d-4f6f-a69d-3366a4e774de-kube-api-access-59bhc\") pod \"service-ca-98c88fbb5-qvjx6\" (UID: \"52f9c956-153d-4f6f-a69d-3366a4e774de\") " pod="openshift-service-ca/service-ca-98c88fbb5-qvjx6"
Aug 02 09:33:32.584003 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:32.583926    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/52f9c956-153d-4f6f-a69d-3366a4e774de-signing-cabundle\") pod \"service-ca-98c88fbb5-qvjx6\" (UID: \"52f9c956-153d-4f6f-a69d-3366a4e774de\") " pod="openshift-service-ca/service-ca-98c88fbb5-qvjx6"
Aug 02 09:33:32.584003 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:32.583979    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/52f9c956-153d-4f6f-a69d-3366a4e774de-signing-key\") pod \"service-ca-98c88fbb5-qvjx6\" (UID: \"52f9c956-153d-4f6f-a69d-3366a4e774de\") " pod="openshift-service-ca/service-ca-98c88fbb5-qvjx6"
Aug 02 09:33:32.584509 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:32.584105    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59bhc\" (UniqueName: \"kubernetes.io/projected/52f9c956-153d-4f6f-a69d-3366a4e774de-kube-api-access-59bhc\") pod \"service-ca-98c88fbb5-qvjx6\" (UID: \"52f9c956-153d-4f6f-a69d-3366a4e774de\") " pod="openshift-service-ca/service-ca-98c88fbb5-qvjx6"
Aug 02 09:33:32.584678 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:32.584651    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/52f9c956-153d-4f6f-a69d-3366a4e774de-signing-cabundle\") pod \"service-ca-98c88fbb5-qvjx6\" (UID: \"52f9c956-153d-4f6f-a69d-3366a4e774de\") " pod="openshift-service-ca/service-ca-98c88fbb5-qvjx6"
Aug 02 09:33:32.586859 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:32.586838    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/52f9c956-153d-4f6f-a69d-3366a4e774de-signing-key\") pod \"service-ca-98c88fbb5-qvjx6\" (UID: \"52f9c956-153d-4f6f-a69d-3366a4e774de\") " pod="openshift-service-ca/service-ca-98c88fbb5-qvjx6"
Aug 02 09:33:32.591647 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:32.591626    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59bhc\" (UniqueName: \"kubernetes.io/projected/52f9c956-153d-4f6f-a69d-3366a4e774de-kube-api-access-59bhc\") pod \"service-ca-98c88fbb5-qvjx6\" (UID: \"52f9c956-153d-4f6f-a69d-3366a4e774de\") " pod="openshift-service-ca/service-ca-98c88fbb5-qvjx6"
Aug 02 09:33:32.689466 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:32.689430    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-98c88fbb5-qvjx6"
Aug 02 09:33:33.221087 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:33.221057    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:33.265514 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:33.265486    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-qvjx6"]
Aug 02 09:33:33.268781 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:33.268759    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pvvv7"]
Aug 02 09:33:33.269334 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:33:33.269301    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52f9c956_153d_4f6f_a69d_3366a4e774de.slice/crio-fe7c26e38c69eac0e0d45168f1fd0b15f74bab8b2ec400ce6271d0dafc34e6b5 WatchSource:0}: Error finding container fe7c26e38c69eac0e0d45168f1fd0b15f74bab8b2ec400ce6271d0dafc34e6b5: Status 404 returned error can't find the container with id fe7c26e38c69eac0e0d45168f1fd0b15f74bab8b2ec400ce6271d0dafc34e6b5
Aug 02 09:33:33.291066 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:33.291039    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/88085931-e03b-4e83-a890-e9b7dfafeda1-registry-tls\") pod \"image-registry-5f6b7788d4-hl9g6\" (UID: \"88085931-e03b-4e83-a890-e9b7dfafeda1\") " pod="openshift-image-registry/image-registry-5f6b7788d4-hl9g6"
Aug 02 09:33:33.291212 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:33.291191    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 02 09:33:33.291212 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:33.291210    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f6b7788d4-hl9g6: secret "image-registry-tls" not found
Aug 02 09:33:33.291353 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:33.291271    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88085931-e03b-4e83-a890-e9b7dfafeda1-registry-tls podName:88085931-e03b-4e83-a890-e9b7dfafeda1 nodeName:}" failed. No retries permitted until 2026-08-02 09:33:35.291256128 +0000 UTC m=+39.006061554 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/88085931-e03b-4e83-a890-e9b7dfafeda1-registry-tls") pod "image-registry-5f6b7788d4-hl9g6" (UID: "88085931-e03b-4e83-a890-e9b7dfafeda1") : secret "image-registry-tls" not found
Aug 02 09:33:33.296181 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:33.296154    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:33.354976 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:33.354933    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:33.391873 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:33.391851    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b6e3c57-2d9d-419a-a7e2-c419d29faecc-metrics-tls\") pod \"dns-default-7f8lc\" (UID: \"6b6e3c57-2d9d-419a-a7e2-c419d29faecc\") " pod="openshift-dns/dns-default-7f8lc"
Aug 02 09:33:33.391966 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:33.391903    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-89bsp\" (UID: \"0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb\") " pod="openshift-insights/insights-runtime-extractor-89bsp"
Aug 02 09:33:33.391966 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:33.391936    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-89bsp\" (UID: \"0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb\") " pod="openshift-insights/insights-runtime-extractor-89bsp"
Aug 02 09:33:33.392074 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:33.392003    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 02 09:33:33.392074 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:33.392036    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 02 09:33:33.392074 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:33.392041    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 02 09:33:33.392074 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:33.392072    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b6e3c57-2d9d-419a-a7e2-c419d29faecc-metrics-tls podName:6b6e3c57-2d9d-419a-a7e2-c419d29faecc nodeName:}" failed. No retries permitted until 2026-08-02 09:33:35.392053076 +0000 UTC m=+39.106858501 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6b6e3c57-2d9d-419a-a7e2-c419d29faecc-metrics-tls") pod "dns-default-7f8lc" (UID: "6b6e3c57-2d9d-419a-a7e2-c419d29faecc") : secret "dns-default-metrics-tls" not found
Aug 02 09:33:33.392221 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:33.392094    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb-kube-rbac-proxy-cm podName:0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb nodeName:}" failed. No retries permitted until 2026-08-02 09:33:35.392078577 +0000 UTC m=+39.106884007 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-89bsp" (UID: "0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb") : configmap "kube-rbac-proxy" not found
Aug 02 09:33:33.392221 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:33.392112    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb-insights-runtime-extractor-tls podName:0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb nodeName:}" failed. No retries permitted until 2026-08-02 09:33:35.392102255 +0000 UTC m=+39.106907687 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-89bsp" (UID: "0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb") : secret "insights-runtime-extractor-tls" not found
Aug 02 09:33:34.142736 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:34.142700    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pvvv7" event={"ID":"80c7ce46-ead1-48ad-9c47-296c0124131d","Type":"ContainerStarted","Data":"13bc36570ac9e91525811caabc519df52cc85d93ec46687171a8842534cc83c3"}
Aug 02 09:33:34.145824 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:34.145793    2564 generic.go:356] "Generic (PLEG): container finished" podID="0fb6399d-1991-4b8e-920c-a1086603b643" containerID="9c1e431deea36aa0ef333a4c11c580f6dc94aefe5e7c3c72ee4f342137f3fa16" exitCode=0
Aug 02 09:33:34.145954 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:34.145892    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qgz2q" event={"ID":"0fb6399d-1991-4b8e-920c-a1086603b643","Type":"ContainerDied","Data":"9c1e431deea36aa0ef333a4c11c580f6dc94aefe5e7c3c72ee4f342137f3fa16"}
Aug 02 09:33:34.147604 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:34.147428    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-98c88fbb5-qvjx6" event={"ID":"52f9c956-153d-4f6f-a69d-3366a4e774de","Type":"ContainerStarted","Data":"fe7c26e38c69eac0e0d45168f1fd0b15f74bab8b2ec400ce6271d0dafc34e6b5"}
Aug 02 09:33:34.782871 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:34.782792    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:35.152194 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:35.152165    2564 generic.go:356] "Generic (PLEG): container finished" podID="0fb6399d-1991-4b8e-920c-a1086603b643" containerID="2cdb2a2601171abeba4045967758a6deede1b2e86c60e953bcb88bebc4602b9b" exitCode=0
Aug 02 09:33:35.152566 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:35.152262    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qgz2q" event={"ID":"0fb6399d-1991-4b8e-920c-a1086603b643","Type":"ContainerDied","Data":"2cdb2a2601171abeba4045967758a6deede1b2e86c60e953bcb88bebc4602b9b"}
Aug 02 09:33:35.308778 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:35.308339    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/88085931-e03b-4e83-a890-e9b7dfafeda1-registry-tls\") pod \"image-registry-5f6b7788d4-hl9g6\" (UID: \"88085931-e03b-4e83-a890-e9b7dfafeda1\") " pod="openshift-image-registry/image-registry-5f6b7788d4-hl9g6"
Aug 02 09:33:35.308778 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:35.308459    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 02 09:33:35.308778 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:35.308476    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f6b7788d4-hl9g6: secret "image-registry-tls" not found
Aug 02 09:33:35.308778 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:35.308536    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88085931-e03b-4e83-a890-e9b7dfafeda1-registry-tls podName:88085931-e03b-4e83-a890-e9b7dfafeda1 nodeName:}" failed. No retries permitted until 2026-08-02 09:33:39.308515531 +0000 UTC m=+43.023320976 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/88085931-e03b-4e83-a890-e9b7dfafeda1-registry-tls") pod "image-registry-5f6b7788d4-hl9g6" (UID: "88085931-e03b-4e83-a890-e9b7dfafeda1") : secret "image-registry-tls" not found
Aug 02 09:33:35.410093 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:35.410003    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b6e3c57-2d9d-419a-a7e2-c419d29faecc-metrics-tls\") pod \"dns-default-7f8lc\" (UID: \"6b6e3c57-2d9d-419a-a7e2-c419d29faecc\") " pod="openshift-dns/dns-default-7f8lc"
Aug 02 09:33:35.410093 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:35.410060    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-89bsp\" (UID: \"0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb\") " pod="openshift-insights/insights-runtime-extractor-89bsp"
Aug 02 09:33:35.410343 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:35.410096    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-89bsp\" (UID: \"0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb\") " pod="openshift-insights/insights-runtime-extractor-89bsp"
Aug 02 09:33:35.410343 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:35.410180    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 02 09:33:35.410343 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:35.410203    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 02 09:33:35.410343 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:35.410273    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b6e3c57-2d9d-419a-a7e2-c419d29faecc-metrics-tls podName:6b6e3c57-2d9d-419a-a7e2-c419d29faecc nodeName:}" failed. No retries permitted until 2026-08-02 09:33:39.410251673 +0000 UTC m=+43.125057099 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6b6e3c57-2d9d-419a-a7e2-c419d29faecc-metrics-tls") pod "dns-default-7f8lc" (UID: "6b6e3c57-2d9d-419a-a7e2-c419d29faecc") : secret "dns-default-metrics-tls" not found
Aug 02 09:33:35.410343 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:35.410294    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb-insights-runtime-extractor-tls podName:0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb nodeName:}" failed. No retries permitted until 2026-08-02 09:33:39.410283714 +0000 UTC m=+43.125089145 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-89bsp" (UID: "0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb") : secret "insights-runtime-extractor-tls" not found
Aug 02 09:33:35.410667 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:35.410646    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-89bsp\" (UID: \"0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb\") " pod="openshift-insights/insights-runtime-extractor-89bsp"
Aug 02 09:33:36.234001 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:36.233973    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:36.643579 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:36.643546    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:36.696811 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:36.696776    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:37.159462 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:37.159427    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pvvv7" event={"ID":"80c7ce46-ead1-48ad-9c47-296c0124131d","Type":"ContainerStarted","Data":"124769d2c8880012a1c93eaa7801b223983c2de66fd3a0034ab5e19c51441762"}
Aug 02 09:33:37.162749 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:37.162723    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qgz2q" event={"ID":"0fb6399d-1991-4b8e-920c-a1086603b643","Type":"ContainerStarted","Data":"8fd7d712308d6bef6806db40d83e692a673edd13dea97353e6e7eca704b8ff9c"}
Aug 02 09:33:37.164090 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:37.164067    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-98c88fbb5-qvjx6" event={"ID":"52f9c956-153d-4f6f-a69d-3366a4e774de","Type":"ContainerStarted","Data":"66bbe54266281a00f5e22f2c9dba86d1e0e1b5180d8cb8dab2a9be83bc4bc773"}
Aug 02 09:33:37.173516 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:37.173469    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-pvvv7" podStartSLOduration=3.108934193 podStartE2EDuration="6.173455826s" totalImagesPullingTime="3.064521633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 09:33:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:33:37.172613528 +0000 UTC m=+40.887418978" watchObservedRunningTime="2026-08-02 09:33:37.173455826 +0000 UTC m=+40.888261274"
Aug 02 09:33:37.193805 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:37.193761    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qgz2q" podStartSLOduration=11.309204441 podStartE2EDuration="40.19374407s" totalImagesPullingTime="28.884539629s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-02 09:32:57 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:33:37.193559707 +0000 UTC m=+40.908365156" watchObservedRunningTime="2026-08-02 09:33:37.19374407 +0000 UTC m=+40.908549521"
Aug 02 09:33:37.207085 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:37.207041    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-98c88fbb5-qvjx6" podStartSLOduration=2.145924429 podStartE2EDuration="5.207024805s" totalImagesPullingTime="3.061100376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 09:33:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:33:37.206201675 +0000 UTC m=+40.921007120" watchObservedRunningTime="2026-08-02 09:33:37.207024805 +0000 UTC m=+40.921830254"
Aug 02 09:33:38.728587 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:38.728552    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:38.777138 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:38.777106    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:39.261925 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:39.261889    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:39.340735 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:39.340702    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/88085931-e03b-4e83-a890-e9b7dfafeda1-registry-tls\") pod \"image-registry-5f6b7788d4-hl9g6\" (UID: \"88085931-e03b-4e83-a890-e9b7dfafeda1\") " pod="openshift-image-registry/image-registry-5f6b7788d4-hl9g6"
Aug 02 09:33:39.340916 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:39.340856    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 02 09:33:39.340916 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:39.340877    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f6b7788d4-hl9g6: secret "image-registry-tls" not found
Aug 02 09:33:39.341046 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:39.340934    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88085931-e03b-4e83-a890-e9b7dfafeda1-registry-tls podName:88085931-e03b-4e83-a890-e9b7dfafeda1 nodeName:}" failed. No retries permitted until 2026-08-02 09:33:47.34091441 +0000 UTC m=+51.055719837 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/88085931-e03b-4e83-a890-e9b7dfafeda1-registry-tls") pod "image-registry-5f6b7788d4-hl9g6" (UID: "88085931-e03b-4e83-a890-e9b7dfafeda1") : secret "image-registry-tls" not found
Aug 02 09:33:39.392445 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:39.392407    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:39.441294 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:39.441269    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b6e3c57-2d9d-419a-a7e2-c419d29faecc-metrics-tls\") pod \"dns-default-7f8lc\" (UID: \"6b6e3c57-2d9d-419a-a7e2-c419d29faecc\") " pod="openshift-dns/dns-default-7f8lc"
Aug 02 09:33:39.441426 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:39.441321    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-89bsp\" (UID: \"0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb\") " pod="openshift-insights/insights-runtime-extractor-89bsp"
Aug 02 09:33:39.441531 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:39.441514    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 02 09:33:39.441637 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:39.441621    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb-insights-runtime-extractor-tls podName:0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb nodeName:}" failed. No retries permitted until 2026-08-02 09:33:47.441603212 +0000 UTC m=+51.156408655 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-89bsp" (UID: "0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb") : secret "insights-runtime-extractor-tls" not found
Aug 02 09:33:39.441709 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:39.441546    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 02 09:33:39.441709 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:33:39.441691    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b6e3c57-2d9d-419a-a7e2-c419d29faecc-metrics-tls podName:6b6e3c57-2d9d-419a-a7e2-c419d29faecc nodeName:}" failed. No retries permitted until 2026-08-02 09:33:47.44168068 +0000 UTC m=+51.156486124 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6b6e3c57-2d9d-419a-a7e2-c419d29faecc-metrics-tls") pod "dns-default-7f8lc" (UID: "6b6e3c57-2d9d-419a-a7e2-c419d29faecc") : secret "dns-default-metrics-tls" not found
Aug 02 09:33:39.495703 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:39.495673    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:39.599466 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:39.599379    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:39.713476 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:39.713453    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:39.796580 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:39.796550    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:39.961083 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:39.961058    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:40.761535 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:40.761503    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:41.357160 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:41.357133    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:41.560677 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:41.560647    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:41.963176 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:41.963147    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:42.157568 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:42.157538    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:42.759378 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:42.759349    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:42.959379 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:42.959352    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:43.144736 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:43.144678    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:43.360535 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:43.360506    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:43.561403 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:43.561328    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:43.761550 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:43.761518    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:43.862015 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:43.861947    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:44.161818 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:44.161794    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:44.761222 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:44.761193    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:45.157810 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:45.157780    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:45.460725 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:45.460651    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:45.561268 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:45.561226    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:45.969306 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:45.969274    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:46.159131 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:46.159103    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:46.557937 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:46.557905    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:47.160294 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:47.160260    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:47.396129 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:47.396099    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/88085931-e03b-4e83-a890-e9b7dfafeda1-registry-tls\") pod \"image-registry-5f6b7788d4-hl9g6\" (UID: \"88085931-e03b-4e83-a890-e9b7dfafeda1\") " pod="openshift-image-registry/image-registry-5f6b7788d4-hl9g6"
Aug 02 09:33:47.399768 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:47.399741    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/88085931-e03b-4e83-a890-e9b7dfafeda1-registry-tls\") pod \"image-registry-5f6b7788d4-hl9g6\" (UID: \"88085931-e03b-4e83-a890-e9b7dfafeda1\") " pod="openshift-image-registry/image-registry-5f6b7788d4-hl9g6"
Aug 02 09:33:47.496446 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:47.496379    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b6e3c57-2d9d-419a-a7e2-c419d29faecc-metrics-tls\") pod \"dns-default-7f8lc\" (UID: \"6b6e3c57-2d9d-419a-a7e2-c419d29faecc\") " pod="openshift-dns/dns-default-7f8lc"
Aug 02 09:33:47.496446 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:47.496426    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-89bsp\" (UID: \"0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb\") " pod="openshift-insights/insights-runtime-extractor-89bsp"
Aug 02 09:33:47.498569 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:47.498540    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b6e3c57-2d9d-419a-a7e2-c419d29faecc-metrics-tls\") pod \"dns-default-7f8lc\" (UID: \"6b6e3c57-2d9d-419a-a7e2-c419d29faecc\") " pod="openshift-dns/dns-default-7f8lc"
Aug 02 09:33:47.498569 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:47.498547    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-89bsp\" (UID: \"0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb\") " pod="openshift-insights/insights-runtime-extractor-89bsp"
Aug 02 09:33:47.507657 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:47.507636    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f6b7788d4-hl9g6"
Aug 02 09:33:47.523926 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:47.523903    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-89bsp"
Aug 02 09:33:47.531571 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:47.531551    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7f8lc"
Aug 02 09:33:47.562599 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:47.562564    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:47.697900 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:47.697866    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f6b7788d4-hl9g6"]
Aug 02 09:33:47.701872 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:33:47.701848    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88085931_e03b_4e83_a890_e9b7dfafeda1.slice/crio-47b60008d63aa32c9ce74e6cf6561ec96d1cf8ce308da08c443f4d96ce185f20 WatchSource:0}: Error finding container 47b60008d63aa32c9ce74e6cf6561ec96d1cf8ce308da08c443f4d96ce185f20: Status 404 returned error can't find the container with id 47b60008d63aa32c9ce74e6cf6561ec96d1cf8ce308da08c443f4d96ce185f20
Aug 02 09:33:47.716212 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:47.716188    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-89bsp"]
Aug 02 09:33:47.719307 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:33:47.719266    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c4922a2_8f81_4ce6_8a51_4f7b727ba0eb.slice/crio-be8f53f513bb03c1b03482ad5b07d053b58dd759e61d73a9e70f052094577a0b WatchSource:0}: Error finding container be8f53f513bb03c1b03482ad5b07d053b58dd759e61d73a9e70f052094577a0b: Status 404 returned error can't find the container with id be8f53f513bb03c1b03482ad5b07d053b58dd759e61d73a9e70f052094577a0b
Aug 02 09:33:47.732224 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:47.732204    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7f8lc"]
Aug 02 09:33:47.750288 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:33:47.750251    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b6e3c57_2d9d_419a_a7e2_c419d29faecc.slice/crio-56ad9103bf12ff2cf0f6e025c9c525b145418b4ba74d36d3a0e25a444b000421 WatchSource:0}: Error finding container 56ad9103bf12ff2cf0f6e025c9c525b145418b4ba74d36d3a0e25a444b000421: Status 404 returned error can't find the container with id 56ad9103bf12ff2cf0f6e025c9c525b145418b4ba74d36d3a0e25a444b000421
Aug 02 09:33:47.961846 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:47.961815    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:48.161955 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:48.161919    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:48.190960 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:48.190927    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7f8lc" event={"ID":"6b6e3c57-2d9d-419a-a7e2-c419d29faecc","Type":"ContainerStarted","Data":"56ad9103bf12ff2cf0f6e025c9c525b145418b4ba74d36d3a0e25a444b000421"}
Aug 02 09:33:48.193017 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:48.192991    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-89bsp" event={"ID":"0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb","Type":"ContainerStarted","Data":"72caf67877814bb0527d8699b591db0d80243f0c9be3d1aa6a80eb4c9392290c"}
Aug 02 09:33:48.193201 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:48.193053    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-89bsp" event={"ID":"0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb","Type":"ContainerStarted","Data":"be8f53f513bb03c1b03482ad5b07d053b58dd759e61d73a9e70f052094577a0b"}
Aug 02 09:33:48.195663 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:48.195635    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f6b7788d4-hl9g6" event={"ID":"88085931-e03b-4e83-a890-e9b7dfafeda1","Type":"ContainerStarted","Data":"f1810bca226cdbf7c7e37b9d5cb1adbd896f5256f93294ae34ef3339b0804a6f"}
Aug 02 09:33:48.195776 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:48.195682    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f6b7788d4-hl9g6" event={"ID":"88085931-e03b-4e83-a890-e9b7dfafeda1","Type":"ContainerStarted","Data":"47b60008d63aa32c9ce74e6cf6561ec96d1cf8ce308da08c443f4d96ce185f20"}
Aug 02 09:33:48.196488 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:48.196466    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5f6b7788d4-hl9g6"
Aug 02 09:33:48.215256 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:48.215203    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5f6b7788d4-hl9g6" podStartSLOduration=34.215184764 podStartE2EDuration="34.215184764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 09:33:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:33:48.213835748 +0000 UTC m=+51.928641199" watchObservedRunningTime="2026-08-02 09:33:48.215184764 +0000 UTC m=+51.929990215"
Aug 02 09:33:48.269435 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:48.269390    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:48.357870 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:48.357835    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:48.762133 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:48.762076    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:49.160664 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:49.160632    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:49.201101 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:49.201069    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-89bsp" event={"ID":"0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb","Type":"ContainerStarted","Data":"de2a5fb32c96fa8ba8c835af45503a28bc7bd3f42bd94c23f6f3835c8c9974dc"}
Aug 02 09:33:49.567166 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:49.567087    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:49.764802 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:49.764772    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:49.962129 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:49.962103    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:50.206699 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:50.206665    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7f8lc" event={"ID":"6b6e3c57-2d9d-419a-a7e2-c419d29faecc","Type":"ContainerStarted","Data":"23b897e658b2043553b1c00d5a131ea5b0f8bc39d6819123c59d91ea59a19ac9"}
Aug 02 09:33:50.840210 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:50.840175    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5f6b7788d4-hl9g6"]
Aug 02 09:33:50.853379 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:50.853347    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-bb47487cb-vx9vr"]
Aug 02 09:33:50.873291 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:50.873267    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55848b7f64-z9qdc"]
Aug 02 09:33:50.873447 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:50.873336    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bb47487cb-vx9vr"
Aug 02 09:33:50.878891 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:50.876859    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Aug 02 09:33:50.878891 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:50.877087    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Aug 02 09:33:50.878891 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:50.877335    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Aug 02 09:33:50.878891 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:50.877557    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Aug 02 09:33:50.891441 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:50.891421    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f7b6b5494-mb45c"]
Aug 02 09:33:50.891575 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:50.891557    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55848b7f64-z9qdc"
Aug 02 09:33:50.893864 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:50.893841    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Aug 02 09:33:50.893969 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:50.893911    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Aug 02 09:33:50.893969 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:50.893853    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Aug 02 09:33:50.893969 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:50.893849    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Aug 02 09:33:50.904908 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:50.904890    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:50.909490 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:50.909476    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f7b6b5494-mb45c"
Aug 02 09:33:50.911647 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:50.911632    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Aug 02 09:33:50.911776 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:50.911762    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-xb5hc\""
Aug 02 09:33:50.913080 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:50.913064    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-bb47487cb-vx9vr"]
Aug 02 09:33:50.913147 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:50.913097    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f7b6b5494-mb45c"]
Aug 02 09:33:50.913147 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:50.913110    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55848b7f64-z9qdc"]
Aug 02 09:33:51.011484 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:51.011457    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:51.027521 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:51.027500    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8wm4\" (UniqueName: \"kubernetes.io/projected/203b048e-5dec-4b26-911b-15e8c5ed3558-kube-api-access-h8wm4\") pod \"cluster-proxy-proxy-agent-55848b7f64-z9qdc\" (UID: \"203b048e-5dec-4b26-911b-15e8c5ed3558\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55848b7f64-z9qdc"
Aug 02 09:33:51.027618 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:51.027527    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/203b048e-5dec-4b26-911b-15e8c5ed3558-hub\") pod \"cluster-proxy-proxy-agent-55848b7f64-z9qdc\" (UID: \"203b048e-5dec-4b26-911b-15e8c5ed3558\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55848b7f64-z9qdc"
Aug 02 09:33:51.027618 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:51.027546    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/203b048e-5dec-4b26-911b-15e8c5ed3558-additional-service-ca\") pod \"cluster-proxy-proxy-agent-55848b7f64-z9qdc\" (UID: \"203b048e-5dec-4b26-911b-15e8c5ed3558\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55848b7f64-z9qdc"
Aug 02 09:33:51.027618 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:51.027564    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kqsg\" (UniqueName: \"kubernetes.io/projected/219f89e8-f8be-4fda-8c2c-b04bb8a5ab5f-kube-api-access-6kqsg\") pod \"managed-serviceaccount-addon-agent-f7b6b5494-mb45c\" (UID: \"219f89e8-f8be-4fda-8c2c-b04bb8a5ab5f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f7b6b5494-mb45c"
Aug 02 09:33:51.027618 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:51.027588    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/219f89e8-f8be-4fda-8c2c-b04bb8a5ab5f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-f7b6b5494-mb45c\" (UID: \"219f89e8-f8be-4fda-8c2c-b04bb8a5ab5f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f7b6b5494-mb45c"
Aug 02 09:33:51.027774 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:51.027647    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/203b048e-5dec-4b26-911b-15e8c5ed3558-ca\") pod \"cluster-proxy-proxy-agent-55848b7f64-z9qdc\" (UID: \"203b048e-5dec-4b26-911b-15e8c5ed3558\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55848b7f64-z9qdc"
Aug 02 09:33:51.027774 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:51.027670    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/203b048e-5dec-4b26-911b-15e8c5ed3558-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-55848b7f64-z9qdc\" (UID: \"203b048e-5dec-4b26-911b-15e8c5ed3558\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55848b7f64-z9qdc"
Aug 02 09:33:51.027774 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:51.027689    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/203b048e-5dec-4b26-911b-15e8c5ed3558-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-55848b7f64-z9qdc\" (UID: \"203b048e-5dec-4b26-911b-15e8c5ed3558\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55848b7f64-z9qdc"
Aug 02 09:33:51.027774 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:51.027706    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/dc6d666a-5ac8-4dab-873c-129892aed9d9-klusterlet-config\") pod \"klusterlet-addon-workmgr-bb47487cb-vx9vr\" (UID: \"dc6d666a-5ac8-4dab-873c-129892aed9d9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bb47487cb-vx9vr"
Aug 02 09:33:51.027912 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:51.027785    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzh25\" (UniqueName: \"kubernetes.io/projected/dc6d666a-5ac8-4dab-873c-129892aed9d9-kube-api-access-xzh25\") pod \"klusterlet-addon-workmgr-bb47487cb-vx9vr\" (UID: \"dc6d666a-5ac8-4dab-873c-129892aed9d9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bb47487cb-vx9vr"
Aug 02 09:33:51.027912 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:51.027818    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc6d666a-5ac8-4dab-873c-129892aed9d9-tmp\") pod \"klusterlet-addon-workmgr-bb47487cb-vx9vr\" (UID: \"dc6d666a-5ac8-4dab-873c-129892aed9d9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bb47487cb-vx9vr"
Aug 02 09:33:51.093622 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:51.093568    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:51.128285 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:51.128268    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/203b048e-5dec-4b26-911b-15e8c5ed3558-ca\") pod \"cluster-proxy-proxy-agent-55848b7f64-z9qdc\" (UID: \"203b048e-5dec-4b26-911b-15e8c5ed3558\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55848b7f64-z9qdc"
Aug 02 09:33:51.128341 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:51.128296    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/203b048e-5dec-4b26-911b-15e8c5ed3558-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-55848b7f64-z9qdc\" (UID: \"203b048e-5dec-4b26-911b-15e8c5ed3558\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55848b7f64-z9qdc"
Aug 02 09:33:51.128341 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:51.128318    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/203b048e-5dec-4b26-911b-15e8c5ed3558-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-55848b7f64-z9qdc\" (UID: \"203b048e-5dec-4b26-911b-15e8c5ed3558\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55848b7f64-z9qdc"
Aug 02 09:33:51.128341 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:51.128340    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/dc6d666a-5ac8-4dab-873c-129892aed9d9-klusterlet-config\") pod \"klusterlet-addon-workmgr-bb47487cb-vx9vr\" (UID: \"dc6d666a-5ac8-4dab-873c-129892aed9d9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bb47487cb-vx9vr"
Aug 02 09:33:51.128553 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:51.128375    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzh25\" (UniqueName: \"kubernetes.io/projected/dc6d666a-5ac8-4dab-873c-129892aed9d9-kube-api-access-xzh25\") pod \"klusterlet-addon-workmgr-bb47487cb-vx9vr\" (UID: \"dc6d666a-5ac8-4dab-873c-129892aed9d9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bb47487cb-vx9vr"
Aug 02 09:33:51.128553 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:51.128402    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc6d666a-5ac8-4dab-873c-129892aed9d9-tmp\") pod \"klusterlet-addon-workmgr-bb47487cb-vx9vr\" (UID: \"dc6d666a-5ac8-4dab-873c-129892aed9d9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bb47487cb-vx9vr"
Aug 02 09:33:51.128553 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:51.128438    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8wm4\" (UniqueName: \"kubernetes.io/projected/203b048e-5dec-4b26-911b-15e8c5ed3558-kube-api-access-h8wm4\") pod \"cluster-proxy-proxy-agent-55848b7f64-z9qdc\" (UID: \"203b048e-5dec-4b26-911b-15e8c5ed3558\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55848b7f64-z9qdc"
Aug 02 09:33:51.128553 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:51.128468    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/203b048e-5dec-4b26-911b-15e8c5ed3558-hub\") pod \"cluster-proxy-proxy-agent-55848b7f64-z9qdc\" (UID: \"203b048e-5dec-4b26-911b-15e8c5ed3558\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55848b7f64-z9qdc"
Aug 02 09:33:51.128553 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:51.128497    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/203b048e-5dec-4b26-911b-15e8c5ed3558-additional-service-ca\") pod \"cluster-proxy-proxy-agent-55848b7f64-z9qdc\" (UID: \"203b048e-5dec-4b26-911b-15e8c5ed3558\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55848b7f64-z9qdc"
Aug 02 09:33:51.128553 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:51.128520    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kqsg\" (UniqueName: \"kubernetes.io/projected/219f89e8-f8be-4fda-8c2c-b04bb8a5ab5f-kube-api-access-6kqsg\") pod \"managed-serviceaccount-addon-agent-f7b6b5494-mb45c\" (UID: \"219f89e8-f8be-4fda-8c2c-b04bb8a5ab5f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f7b6b5494-mb45c"
Aug 02 09:33:51.128553 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:51.128543    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/219f89e8-f8be-4fda-8c2c-b04bb8a5ab5f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-f7b6b5494-mb45c\" (UID: \"219f89e8-f8be-4fda-8c2c-b04bb8a5ab5f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f7b6b5494-mb45c"
Aug 02 09:33:51.129316 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:51.129275    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc6d666a-5ac8-4dab-873c-129892aed9d9-tmp\") pod \"klusterlet-addon-workmgr-bb47487cb-vx9vr\" (UID: \"dc6d666a-5ac8-4dab-873c-129892aed9d9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bb47487cb-vx9vr"
Aug 02 09:33:51.129458 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:51.129438    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/203b048e-5dec-4b26-911b-15e8c5ed3558-additional-service-ca\") pod \"cluster-proxy-proxy-agent-55848b7f64-z9qdc\" (UID: \"203b048e-5dec-4b26-911b-15e8c5ed3558\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55848b7f64-z9qdc"
Aug 02 09:33:51.131392 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:51.131367    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/203b048e-5dec-4b26-911b-15e8c5ed3558-ca\") pod \"cluster-proxy-proxy-agent-55848b7f64-z9qdc\" (UID: \"203b048e-5dec-4b26-911b-15e8c5ed3558\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55848b7f64-z9qdc"
Aug 02 09:33:51.131479 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:51.131413    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/219f89e8-f8be-4fda-8c2c-b04bb8a5ab5f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-f7b6b5494-mb45c\" (UID: \"219f89e8-f8be-4fda-8c2c-b04bb8a5ab5f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f7b6b5494-mb45c"
Aug 02 09:33:51.131479 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:51.131417    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/dc6d666a-5ac8-4dab-873c-129892aed9d9-klusterlet-config\") pod \"klusterlet-addon-workmgr-bb47487cb-vx9vr\" (UID: \"dc6d666a-5ac8-4dab-873c-129892aed9d9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bb47487cb-vx9vr"
Aug 02 09:33:51.131588 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:51.131569    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/203b048e-5dec-4b26-911b-15e8c5ed3558-hub\") pod \"cluster-proxy-proxy-agent-55848b7f64-z9qdc\" (UID: \"203b048e-5dec-4b26-911b-15e8c5ed3558\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55848b7f64-z9qdc"
Aug 02 09:33:51.131626 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:51.131617    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/203b048e-5dec-4b26-911b-15e8c5ed3558-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-55848b7f64-z9qdc\" (UID: \"203b048e-5dec-4b26-911b-15e8c5ed3558\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55848b7f64-z9qdc"
Aug 02 09:33:51.131725 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:51.131710    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/203b048e-5dec-4b26-911b-15e8c5ed3558-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-55848b7f64-z9qdc\" (UID: \"203b048e-5dec-4b26-911b-15e8c5ed3558\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55848b7f64-z9qdc"
Aug 02 09:33:51.140646 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:51.140624    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kqsg\" (UniqueName: \"kubernetes.io/projected/219f89e8-f8be-4fda-8c2c-b04bb8a5ab5f-kube-api-access-6kqsg\") pod \"managed-serviceaccount-addon-agent-f7b6b5494-mb45c\" (UID: \"219f89e8-f8be-4fda-8c2c-b04bb8a5ab5f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f7b6b5494-mb45c"
Aug 02 09:33:51.140646 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:51.140642    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8wm4\" (UniqueName: \"kubernetes.io/projected/203b048e-5dec-4b26-911b-15e8c5ed3558-kube-api-access-h8wm4\") pod \"cluster-proxy-proxy-agent-55848b7f64-z9qdc\" (UID: \"203b048e-5dec-4b26-911b-15e8c5ed3558\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55848b7f64-z9qdc"
Aug 02 09:33:51.141031 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:51.141008    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzh25\" (UniqueName: \"kubernetes.io/projected/dc6d666a-5ac8-4dab-873c-129892aed9d9-kube-api-access-xzh25\") pod \"klusterlet-addon-workmgr-bb47487cb-vx9vr\" (UID: \"dc6d666a-5ac8-4dab-873c-129892aed9d9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bb47487cb-vx9vr"
Aug 02 09:33:51.186171 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:51.186153    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bb47487cb-vx9vr"
Aug 02 09:33:51.187491 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:51.187470    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:51.211465 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:51.211423    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7f8lc" event={"ID":"6b6e3c57-2d9d-419a-a7e2-c419d29faecc","Type":"ContainerStarted","Data":"a98aacf04a35560abf992017bec3c01233740a4d430e8eaef601080219c06930"}
Aug 02 09:33:51.218555 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:51.218533    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55848b7f64-z9qdc"
Aug 02 09:33:51.225409 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:51.225388    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f7b6b5494-mb45c"
Aug 02 09:33:51.281580 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:51.281528    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-7f8lc" podStartSLOduration=18.084824275 podStartE2EDuration="20.281506875s" totalImagesPullingTime="2.1966826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 09:33:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:33:51.28051838 +0000 UTC m=+54.995323830" watchObservedRunningTime="2026-08-02 09:33:51.281506875 +0000 UTC m=+54.996312325"
Aug 02 09:33:51.331854 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:51.331635    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-bb47487cb-vx9vr"]
Aug 02 09:33:51.333917 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:33:51.333800    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc6d666a_5ac8_4dab_873c_129892aed9d9.slice/crio-456255812dd7b4391f49e9f5e756884eb260bc2dc7f362ff5351589ac5c39d3d WatchSource:0}: Error finding container 456255812dd7b4391f49e9f5e756884eb260bc2dc7f362ff5351589ac5c39d3d: Status 404 returned error can't find the container with id 456255812dd7b4391f49e9f5e756884eb260bc2dc7f362ff5351589ac5c39d3d
Aug 02 09:33:51.363428 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:51.363368    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55848b7f64-z9qdc"]
Aug 02 09:33:51.366584 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:33:51.366557    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod203b048e_5dec_4b26_911b_15e8c5ed3558.slice/crio-c1684536b5d673f224ae14ab25e3933167032b61ae8f3fef52e59cf389ac8a20 WatchSource:0}: Error finding container c1684536b5d673f224ae14ab25e3933167032b61ae8f3fef52e59cf389ac8a20: Status 404 returned error can't find the container with id c1684536b5d673f224ae14ab25e3933167032b61ae8f3fef52e59cf389ac8a20
Aug 02 09:33:51.376731 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:51.376680    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f7b6b5494-mb45c"]
Aug 02 09:33:51.379511 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:33:51.379490    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod219f89e8_f8be_4fda_8c2c_b04bb8a5ab5f.slice/crio-d6857952bafd716786276d65dfa94b2ff3e0da240f25b063398feea48daad361 WatchSource:0}: Error finding container d6857952bafd716786276d65dfa94b2ff3e0da240f25b063398feea48daad361: Status 404 returned error can't find the container with id d6857952bafd716786276d65dfa94b2ff3e0da240f25b063398feea48daad361
Aug 02 09:33:51.843350 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:51.843310    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:52.219099 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:52.219063    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-89bsp" event={"ID":"0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb","Type":"ContainerStarted","Data":"6a1b22e84c9009d70ca512b1e5f8aa0e578bb610085d7d0af3333adf8023e48a"}
Aug 02 09:33:52.220850 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:52.220818    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55848b7f64-z9qdc" event={"ID":"203b048e-5dec-4b26-911b-15e8c5ed3558","Type":"ContainerStarted","Data":"c1684536b5d673f224ae14ab25e3933167032b61ae8f3fef52e59cf389ac8a20"}
Aug 02 09:33:52.222730 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:52.222689    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f7b6b5494-mb45c" event={"ID":"219f89e8-f8be-4fda-8c2c-b04bb8a5ab5f","Type":"ContainerStarted","Data":"d6857952bafd716786276d65dfa94b2ff3e0da240f25b063398feea48daad361"}
Aug 02 09:33:52.225336 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:52.225254    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bb47487cb-vx9vr" event={"ID":"dc6d666a-5ac8-4dab-873c-129892aed9d9","Type":"ContainerStarted","Data":"456255812dd7b4391f49e9f5e756884eb260bc2dc7f362ff5351589ac5c39d3d"}
Aug 02 09:33:52.225336 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:52.225296    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7f8lc"
Aug 02 09:33:52.234451 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:52.234374    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:52.238803 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:52.237697    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-89bsp" podStartSLOduration=18.053300033 podStartE2EDuration="21.237681369s" totalImagesPullingTime="3.184381336s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-02 09:33:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:33:52.237438649 +0000 UTC m=+55.952244092" watchObservedRunningTime="2026-08-02 09:33:52.237681369 +0000 UTC m=+55.952486818"
Aug 02 09:33:52.334493 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:52.334460    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:56.537689 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:56.537661    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:56.640059 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:56.640021    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:56.737149 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:56.737115    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:56.815762 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:56.815691    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:56.870908 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:56.870879    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:58.135604 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:58.135582    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-59x9w"
Aug 02 09:33:58.284022 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:58.283921    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:58.382537 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:58.382511    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:59.251630 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:59.251596    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f7b6b5494-mb45c" event={"ID":"219f89e8-f8be-4fda-8c2c-b04bb8a5ab5f","Type":"ContainerStarted","Data":"59889a785ced817413c6293a994d78a6ef452df94cc4dd32f091cb0bc3cddbef"}
Aug 02 09:33:59.253225 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:59.253195    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bb47487cb-vx9vr" event={"ID":"dc6d666a-5ac8-4dab-873c-129892aed9d9","Type":"ContainerStarted","Data":"6b91b869a8cca3d00cdaf136499eef6fbcbabb3c8ce857e2c45aa443c764f4f6"}
Aug 02 09:33:59.253384 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:59.253346    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bb47487cb-vx9vr"
Aug 02 09:33:59.255175 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:59.255152    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bb47487cb-vx9vr"
Aug 02 09:33:59.255390 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:59.255365    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55848b7f64-z9qdc" event={"ID":"203b048e-5dec-4b26-911b-15e8c5ed3558","Type":"ContainerStarted","Data":"a15098c79bd1aab4b8534e0768c8eb22ef4b46bb921f5d2f09a5a15095550b8c"}
Aug 02 09:33:59.255390 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:59.255399    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55848b7f64-z9qdc" event={"ID":"203b048e-5dec-4b26-911b-15e8c5ed3558","Type":"ContainerStarted","Data":"e3e9885decb0e0994dd33809fa9b114852fe83ab9bc9cfe6b775387ea2d59c05"}
Aug 02 09:33:59.255534 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:59.255411    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55848b7f64-z9qdc" event={"ID":"203b048e-5dec-4b26-911b-15e8c5ed3558","Type":"ContainerStarted","Data":"645d15fe20f8b99b0449c69874f575d19a73149d0589c08f1c6aaecbb7c859b3"}
Aug 02 09:33:59.264483 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:59.264438    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f7b6b5494-mb45c" podStartSLOduration=2.456080227 podStartE2EDuration="9.264425735s" totalImagesPullingTime="6.808345508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 09:33:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:33:59.263712278 +0000 UTC m=+62.978517727" watchObservedRunningTime="2026-08-02 09:33:59.264425735 +0000 UTC m=+62.979231180"
Aug 02 09:33:59.279655 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:59.279607    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bb47487cb-vx9vr" podStartSLOduration=2.415501023 podStartE2EDuration="9.279594839s" totalImagesPullingTime="6.864093816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 09:33:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:33:59.279111925 +0000 UTC m=+62.993917374" watchObservedRunningTime="2026-08-02 09:33:59.279594839 +0000 UTC m=+62.994400286"
Aug 02 09:33:59.296916 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:59.296867    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55848b7f64-z9qdc" podStartSLOduration=2.483794315 podStartE2EDuration="9.296848227s" totalImagesPullingTime="6.813053912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 09:33:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:33:59.295818355 +0000 UTC m=+63.010623804" watchObservedRunningTime="2026-08-02 09:33:59.296848227 +0000 UTC m=+63.011653676"
Aug 02 09:33:59.832408 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:59.832378    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:33:59.932983 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:33:59.932955    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:00.220183 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:00.220156    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:01.255190 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:01.255132    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:01.355486 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:01.355454    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:02.231939 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:02.231914    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7f8lc"
Aug 02 09:34:02.309689 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:02.309650    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:02.524628 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:02.524539    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2068f737-ee2f-4310-977e-27cb7e1d4f73-metrics-certs\") pod \"network-metrics-daemon-v5mrn\" (UID: \"2068f737-ee2f-4310-977e-27cb7e1d4f73\") " pod="openshift-multus/network-metrics-daemon-v5mrn"
Aug 02 09:34:02.526666 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:02.526648    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Aug 02 09:34:02.537486 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:02.537464    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2068f737-ee2f-4310-977e-27cb7e1d4f73-metrics-certs\") pod \"network-metrics-daemon-v5mrn\" (UID: \"2068f737-ee2f-4310-977e-27cb7e1d4f73\") " pod="openshift-multus/network-metrics-daemon-v5mrn"
Aug 02 09:34:02.625518 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:02.625495    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e6944ae2-aaed-4a4a-9c46-dda99772fa54-original-pull-secret\") pod \"global-pull-secret-syncer-cs5vh\" (UID: \"e6944ae2-aaed-4a4a-9c46-dda99772fa54\") " pod="kube-system/global-pull-secret-syncer-cs5vh"
Aug 02 09:34:02.627975 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:02.627959    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Aug 02 09:34:02.637772 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:02.637756    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e6944ae2-aaed-4a4a-9c46-dda99772fa54-original-pull-secret\") pod \"global-pull-secret-syncer-cs5vh\" (UID: \"e6944ae2-aaed-4a4a-9c46-dda99772fa54\") " pod="kube-system/global-pull-secret-syncer-cs5vh"
Aug 02 09:34:02.726814 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:02.726777    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtx9x\" (UniqueName: \"kubernetes.io/projected/3c47caf2-5d96-4343-ab9a-5da7195e209c-kube-api-access-mtx9x\") pod \"network-check-target-m9jmd\" (UID: \"3c47caf2-5d96-4343-ab9a-5da7195e209c\") " pod="openshift-network-diagnostics/network-check-target-m9jmd"
Aug 02 09:34:02.728890 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:02.728874    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Aug 02 09:34:02.739187 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:02.739167    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Aug 02 09:34:02.750055 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:02.750011    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtx9x\" (UniqueName: \"kubernetes.io/projected/3c47caf2-5d96-4343-ab9a-5da7195e209c-kube-api-access-mtx9x\") pod \"network-check-target-m9jmd\" (UID: \"3c47caf2-5d96-4343-ab9a-5da7195e209c\") " pod="openshift-network-diagnostics/network-check-target-m9jmd"
Aug 02 09:34:02.803280 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:02.803192    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mbl9n\""
Aug 02 09:34:02.810633 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:02.810616    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xt8nz\""
Aug 02 09:34:02.811679 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:02.811665    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9jmd"
Aug 02 09:34:02.816099 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:02.816085    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cs5vh"
Aug 02 09:34:02.819653 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:02.819629    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v5mrn"
Aug 02 09:34:02.953567 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:02.953502    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-m9jmd"]
Aug 02 09:34:02.956741 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:34:02.956704    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c47caf2_5d96_4343_ab9a_5da7195e209c.slice/crio-aa3bff196ba755e46b06238942bf0df60ebfa421239151abf9d92cf3dbbd3ed0 WatchSource:0}: Error finding container aa3bff196ba755e46b06238942bf0df60ebfa421239151abf9d92cf3dbbd3ed0: Status 404 returned error can't find the container with id aa3bff196ba755e46b06238942bf0df60ebfa421239151abf9d92cf3dbbd3ed0
Aug 02 09:34:02.973296 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:02.973271    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-v5mrn"]
Aug 02 09:34:02.976205 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:34:02.976178    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2068f737_ee2f_4310_977e_27cb7e1d4f73.slice/crio-1dceb5721ae55283e0669fb187618479bd16dbc0047a7374a8c7c8607d6cbcff WatchSource:0}: Error finding container 1dceb5721ae55283e0669fb187618479bd16dbc0047a7374a8c7c8607d6cbcff: Status 404 returned error can't find the container with id 1dceb5721ae55283e0669fb187618479bd16dbc0047a7374a8c7c8607d6cbcff
Aug 02 09:34:02.991286 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:02.991265    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cs5vh"]
Aug 02 09:34:02.994182 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:34:02.994161    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6944ae2_aaed_4a4a_9c46_dda99772fa54.slice/crio-45c2bb27b204ac68934a8864440e73ba5550bd4601c4a83b13be48ca2d993176 WatchSource:0}: Error finding container 45c2bb27b204ac68934a8864440e73ba5550bd4601c4a83b13be48ca2d993176: Status 404 returned error can't find the container with id 45c2bb27b204ac68934a8864440e73ba5550bd4601c4a83b13be48ca2d993176
Aug 02 09:34:03.266346 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:03.266315    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:03.268816 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:03.268789    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cs5vh" event={"ID":"e6944ae2-aaed-4a4a-9c46-dda99772fa54","Type":"ContainerStarted","Data":"45c2bb27b204ac68934a8864440e73ba5550bd4601c4a83b13be48ca2d993176"}
Aug 02 09:34:03.269959 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:03.269937    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v5mrn" event={"ID":"2068f737-ee2f-4310-977e-27cb7e1d4f73","Type":"ContainerStarted","Data":"1dceb5721ae55283e0669fb187618479bd16dbc0047a7374a8c7c8607d6cbcff"}
Aug 02 09:34:03.271086 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:03.271064    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-m9jmd" event={"ID":"3c47caf2-5d96-4343-ab9a-5da7195e209c","Type":"ContainerStarted","Data":"aa3bff196ba755e46b06238942bf0df60ebfa421239151abf9d92cf3dbbd3ed0"}
Aug 02 09:34:03.366219 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:03.366188    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:03.976533 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:03.976435    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:04.187403 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:04.187371    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:04.237413 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:04.237339    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-88xrg"]
Aug 02 09:34:04.250866 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:04.250846    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-88xrg"
Aug 02 09:34:04.254209 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:04.254190    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Aug 02 09:34:04.254477 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:04.254461    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Aug 02 09:34:04.254758 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:04.254743    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Aug 02 09:34:04.254992 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:04.254974    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Aug 02 09:34:04.255264 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:04.255221    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Aug 02 09:34:04.255461 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:04.255445    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vw8xp\""
Aug 02 09:34:04.255611 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:04.254974    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Aug 02 09:34:04.259136 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:04.259110    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:04.314800 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:04.314774    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:04.339592 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:04.339566    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztmxb\" (UniqueName: \"kubernetes.io/projected/b8b72c7d-38e1-4478-ab62-465e91cf5434-kube-api-access-ztmxb\") pod \"node-exporter-88xrg\" (UID: \"b8b72c7d-38e1-4478-ab62-465e91cf5434\") " pod="openshift-monitoring/node-exporter-88xrg"
Aug 02 09:34:04.339724 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:04.339599    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b8b72c7d-38e1-4478-ab62-465e91cf5434-node-exporter-wtmp\") pod \"node-exporter-88xrg\" (UID: \"b8b72c7d-38e1-4478-ab62-465e91cf5434\") " pod="openshift-monitoring/node-exporter-88xrg"
Aug 02 09:34:04.339724 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:04.339618    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8b72c7d-38e1-4478-ab62-465e91cf5434-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-88xrg\" (UID: \"b8b72c7d-38e1-4478-ab62-465e91cf5434\") " pod="openshift-monitoring/node-exporter-88xrg"
Aug 02 09:34:04.339724 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:04.339643    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b8b72c7d-38e1-4478-ab62-465e91cf5434-root\") pod \"node-exporter-88xrg\" (UID: \"b8b72c7d-38e1-4478-ab62-465e91cf5434\") " pod="openshift-monitoring/node-exporter-88xrg"
Aug 02 09:34:04.339724 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:04.339681    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b8b72c7d-38e1-4478-ab62-465e91cf5434-node-exporter-textfile\") pod \"node-exporter-88xrg\" (UID: \"b8b72c7d-38e1-4478-ab62-465e91cf5434\") " pod="openshift-monitoring/node-exporter-88xrg"
Aug 02 09:34:04.339955 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:04.339726    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8b72c7d-38e1-4478-ab62-465e91cf5434-metrics-client-ca\") pod \"node-exporter-88xrg\" (UID: \"b8b72c7d-38e1-4478-ab62-465e91cf5434\") " pod="openshift-monitoring/node-exporter-88xrg"
Aug 02 09:34:04.339955 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:04.339763    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b8b72c7d-38e1-4478-ab62-465e91cf5434-node-exporter-accelerators-collector-config\") pod \"node-exporter-88xrg\" (UID: \"b8b72c7d-38e1-4478-ab62-465e91cf5434\") " pod="openshift-monitoring/node-exporter-88xrg"
Aug 02 09:34:04.339955 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:04.339836    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b8b72c7d-38e1-4478-ab62-465e91cf5434-sys\") pod \"node-exporter-88xrg\" (UID: \"b8b72c7d-38e1-4478-ab62-465e91cf5434\") " pod="openshift-monitoring/node-exporter-88xrg"
Aug 02 09:34:04.339955 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:04.339868    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b8b72c7d-38e1-4478-ab62-465e91cf5434-node-exporter-tls\") pod \"node-exporter-88xrg\" (UID: \"b8b72c7d-38e1-4478-ab62-465e91cf5434\") " pod="openshift-monitoring/node-exporter-88xrg"
Aug 02 09:34:04.416143 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:04.416110    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:04.440689 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:04.440659    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztmxb\" (UniqueName: \"kubernetes.io/projected/b8b72c7d-38e1-4478-ab62-465e91cf5434-kube-api-access-ztmxb\") pod \"node-exporter-88xrg\" (UID: \"b8b72c7d-38e1-4478-ab62-465e91cf5434\") " pod="openshift-monitoring/node-exporter-88xrg"
Aug 02 09:34:04.440827 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:04.440702    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b8b72c7d-38e1-4478-ab62-465e91cf5434-node-exporter-wtmp\") pod \"node-exporter-88xrg\" (UID: \"b8b72c7d-38e1-4478-ab62-465e91cf5434\") " pod="openshift-monitoring/node-exporter-88xrg"
Aug 02 09:34:04.440827 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:04.440728    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8b72c7d-38e1-4478-ab62-465e91cf5434-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-88xrg\" (UID: \"b8b72c7d-38e1-4478-ab62-465e91cf5434\") " pod="openshift-monitoring/node-exporter-88xrg"
Aug 02 09:34:04.440827 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:04.440757    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b8b72c7d-38e1-4478-ab62-465e91cf5434-root\") pod \"node-exporter-88xrg\" (UID: \"b8b72c7d-38e1-4478-ab62-465e91cf5434\") " pod="openshift-monitoring/node-exporter-88xrg"
Aug 02 09:34:04.440827 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:04.440789    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b8b72c7d-38e1-4478-ab62-465e91cf5434-node-exporter-textfile\") pod \"node-exporter-88xrg\" (UID: \"b8b72c7d-38e1-4478-ab62-465e91cf5434\") " pod="openshift-monitoring/node-exporter-88xrg"
Aug 02 09:34:04.440827 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:04.440820    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8b72c7d-38e1-4478-ab62-465e91cf5434-metrics-client-ca\") pod \"node-exporter-88xrg\" (UID: \"b8b72c7d-38e1-4478-ab62-465e91cf5434\") " pod="openshift-monitoring/node-exporter-88xrg"
Aug 02 09:34:04.441085 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:04.440850    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b8b72c7d-38e1-4478-ab62-465e91cf5434-node-exporter-accelerators-collector-config\") pod \"node-exporter-88xrg\" (UID: \"b8b72c7d-38e1-4478-ab62-465e91cf5434\") " pod="openshift-monitoring/node-exporter-88xrg"
Aug 02 09:34:04.441085 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:04.440890    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b8b72c7d-38e1-4478-ab62-465e91cf5434-sys\") pod \"node-exporter-88xrg\" (UID: \"b8b72c7d-38e1-4478-ab62-465e91cf5434\") " pod="openshift-monitoring/node-exporter-88xrg"
Aug 02 09:34:04.441085 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:04.440919    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b8b72c7d-38e1-4478-ab62-465e91cf5434-node-exporter-tls\") pod \"node-exporter-88xrg\" (UID: \"b8b72c7d-38e1-4478-ab62-465e91cf5434\") " pod="openshift-monitoring/node-exporter-88xrg"
Aug 02 09:34:04.441255 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:04.441179    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b8b72c7d-38e1-4478-ab62-465e91cf5434-node-exporter-textfile\") pod \"node-exporter-88xrg\" (UID: \"b8b72c7d-38e1-4478-ab62-465e91cf5434\") " pod="openshift-monitoring/node-exporter-88xrg"
Aug 02 09:34:04.441772 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:04.441527    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8b72c7d-38e1-4478-ab62-465e91cf5434-metrics-client-ca\") pod \"node-exporter-88xrg\" (UID: \"b8b72c7d-38e1-4478-ab62-465e91cf5434\") " pod="openshift-monitoring/node-exporter-88xrg"
Aug 02 09:34:04.441772 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:04.441620    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b8b72c7d-38e1-4478-ab62-465e91cf5434-sys\") pod \"node-exporter-88xrg\" (UID: \"b8b72c7d-38e1-4478-ab62-465e91cf5434\") " pod="openshift-monitoring/node-exporter-88xrg"
Aug 02 09:34:04.441772 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:04.441655    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b8b72c7d-38e1-4478-ab62-465e91cf5434-root\") pod \"node-exporter-88xrg\" (UID: \"b8b72c7d-38e1-4478-ab62-465e91cf5434\") " pod="openshift-monitoring/node-exporter-88xrg"
Aug 02 09:34:04.441991 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:04.441778    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b8b72c7d-38e1-4478-ab62-465e91cf5434-node-exporter-wtmp\") pod \"node-exporter-88xrg\" (UID: \"b8b72c7d-38e1-4478-ab62-465e91cf5434\") " pod="openshift-monitoring/node-exporter-88xrg"
Aug 02 09:34:04.442174 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:04.442150    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b8b72c7d-38e1-4478-ab62-465e91cf5434-node-exporter-accelerators-collector-config\") pod \"node-exporter-88xrg\" (UID: \"b8b72c7d-38e1-4478-ab62-465e91cf5434\") " pod="openshift-monitoring/node-exporter-88xrg"
Aug 02 09:34:04.449148 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:04.449106    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztmxb\" (UniqueName: \"kubernetes.io/projected/b8b72c7d-38e1-4478-ab62-465e91cf5434-kube-api-access-ztmxb\") pod \"node-exporter-88xrg\" (UID: \"b8b72c7d-38e1-4478-ab62-465e91cf5434\") " pod="openshift-monitoring/node-exporter-88xrg"
Aug 02 09:34:04.453583 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:04.453557    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8b72c7d-38e1-4478-ab62-465e91cf5434-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-88xrg\" (UID: \"b8b72c7d-38e1-4478-ab62-465e91cf5434\") " pod="openshift-monitoring/node-exporter-88xrg"
Aug 02 09:34:04.454660 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:04.454613    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b8b72c7d-38e1-4478-ab62-465e91cf5434-node-exporter-tls\") pod \"node-exporter-88xrg\" (UID: \"b8b72c7d-38e1-4478-ab62-465e91cf5434\") " pod="openshift-monitoring/node-exporter-88xrg"
Aug 02 09:34:04.515489 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:04.515461    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:04.565873 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:04.565673    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-88xrg"
Aug 02 09:34:04.615006 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:04.614915    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:04.715279 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:04.715185    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:05.054971 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:05.054866    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:05.204425 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:05.204392    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:05.282366 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:05.282320    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:05.284026 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:05.283991    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v5mrn" event={"ID":"2068f737-ee2f-4310-977e-27cb7e1d4f73","Type":"ContainerStarted","Data":"044007a39a8d20d2863f12866f5a8c8e4c1f5c1e740c985ac5569f5732a9ecb9"}
Aug 02 09:34:05.284168 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:05.284094    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v5mrn" event={"ID":"2068f737-ee2f-4310-977e-27cb7e1d4f73","Type":"ContainerStarted","Data":"fa3c2f9f9403cf8ce1340dfecea50402e3416ec8eb54bc241d5ba00f2ceed905"}
Aug 02 09:34:05.288100 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:05.288043    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-88xrg" event={"ID":"b8b72c7d-38e1-4478-ab62-465e91cf5434","Type":"ContainerStarted","Data":"5eb65bbfb6a5dfe14a7747c8d2868020d828c2b6c556ef5c847bee999119686a"}
Aug 02 09:34:05.307996 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:05.307485    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-v5mrn" podStartSLOduration=66.790654103 podStartE2EDuration="1m8.307466583s" totalImagesPullingTime="1.51681248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 09:32:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:34:05.304984352 +0000 UTC m=+69.019789802" watchObservedRunningTime="2026-08-02 09:34:05.307466583 +0000 UTC m=+69.022272033"
Aug 02 09:34:05.410922 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:05.410863    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:05.510187 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:05.510139    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:07.138759 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:07.138664    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:07.187334 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:07.187295    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:07.340190 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:07.340152    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:07.429522 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:07.429439    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:07.636466 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:07.636429    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:07.737134 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:07.737063    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:07.838044 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:07.838015    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:08.409503 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:08.409458    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:08.511743 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:08.511716    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:08.993430 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:08.993398    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:09.304512 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:09.304418    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cs5vh" event={"ID":"e6944ae2-aaed-4a4a-9c46-dda99772fa54","Type":"ContainerStarted","Data":"8adf3c16433c715425c6be8e75c15e6f1c7f0e0c16c9ed8628a12657ec0cfbbe"}
Aug 02 09:34:09.306122 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:09.306100    2564 generic.go:356] "Generic (PLEG): container finished" podID="b8b72c7d-38e1-4478-ab62-465e91cf5434" containerID="831b4f8ec1673ea69524bc8b071588f53f54cf4d5caa789d3b9b4b5357fa6f4d" exitCode=0
Aug 02 09:34:09.306252 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:09.306188    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-88xrg" event={"ID":"b8b72c7d-38e1-4478-ab62-465e91cf5434","Type":"ContainerDied","Data":"831b4f8ec1673ea69524bc8b071588f53f54cf4d5caa789d3b9b4b5357fa6f4d"}
Aug 02 09:34:09.307734 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:09.307714    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-m9jmd" event={"ID":"3c47caf2-5d96-4343-ab9a-5da7195e209c","Type":"ContainerStarted","Data":"de4373866dc7e4d6d185fcd513e184eab2703a023a7e26b26a4f9194d5bba26b"}
Aug 02 09:34:09.307868 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:09.307856    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-m9jmd"
Aug 02 09:34:09.318018 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:09.317979    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-cs5vh" podStartSLOduration=66.869665043 podStartE2EDuration="1m12.317965873s" totalImagesPullingTime="5.44830083s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-02 09:32:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:34:09.317607299 +0000 UTC m=+73.032412751" watchObservedRunningTime="2026-08-02 09:34:09.317965873 +0000 UTC m=+73.032771322"
Aug 02 09:34:09.346738 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:09.346692    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-m9jmd" podStartSLOduration=66.873479102 podStartE2EDuration="1m12.346679406s" totalImagesPullingTime="5.473200304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 09:32:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:34:09.345936083 +0000 UTC m=+73.060741537" watchObservedRunningTime="2026-08-02 09:34:09.346679406 +0000 UTC m=+73.061484851"
Aug 02 09:34:09.618414 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:09.618314    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:10.313247 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:10.313198    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-88xrg" event={"ID":"b8b72c7d-38e1-4478-ab62-465e91cf5434","Type":"ContainerStarted","Data":"546789da7a9fc1f3fbd3fec4d728ea4eb9322afbcd6e0ca4794b89ff091b0660"}
Aug 02 09:34:10.313412 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:10.313268    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-88xrg" event={"ID":"b8b72c7d-38e1-4478-ab62-465e91cf5434","Type":"ContainerStarted","Data":"fdc55a5f184ae7ffb3f75d9a4d7ee14d75ef89b988b5b588a0a020f876635ce5"}
Aug 02 09:34:10.333886 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:10.333839    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-88xrg" podStartSLOduration=2.445549039 podStartE2EDuration="6.333822495s" totalImagesPullingTime="3.816542039s" totalInitContainerRuntime="71.731417ms" isStatefulPod=true podCreationTimestamp="2026-08-02 09:34:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:34:10.3322248 +0000 UTC m=+74.047030247" watchObservedRunningTime="2026-08-02 09:34:10.333822495 +0000 UTC m=+74.048627942"
Aug 02 09:34:10.384977 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:10.384941    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:11.217098 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:11.217070    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5f6b7788d4-hl9g6"
Aug 02 09:34:11.345065 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:11.345017    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:11.397066 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:11.397046    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:14.329514 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:14.329482    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:14.425711 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:14.425682    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:16.229154 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:16.229111    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5f6b7788d4-hl9g6" podUID="88085931-e03b-4e83-a890-e9b7dfafeda1" containerName="registry" containerID="cri-o://f1810bca226cdbf7c7e37b9d5cb1adbd896f5256f93294ae34ef3339b0804a6f" gracePeriod=30
Aug 02 09:34:16.468227 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:16.468205    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f6b7788d4-hl9g6"
Aug 02 09:34:16.530029 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:16.529962    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/88085931-e03b-4e83-a890-e9b7dfafeda1-installation-pull-secrets\") pod \"88085931-e03b-4e83-a890-e9b7dfafeda1\" (UID: \"88085931-e03b-4e83-a890-e9b7dfafeda1\") "
Aug 02 09:34:16.530029 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:16.529996    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/88085931-e03b-4e83-a890-e9b7dfafeda1-bound-sa-token\") pod \"88085931-e03b-4e83-a890-e9b7dfafeda1\" (UID: \"88085931-e03b-4e83-a890-e9b7dfafeda1\") "
Aug 02 09:34:16.530263 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:16.530033    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/88085931-e03b-4e83-a890-e9b7dfafeda1-ca-trust-extracted\") pod \"88085931-e03b-4e83-a890-e9b7dfafeda1\" (UID: \"88085931-e03b-4e83-a890-e9b7dfafeda1\") "
Aug 02 09:34:16.530263 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:16.530063    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/88085931-e03b-4e83-a890-e9b7dfafeda1-trusted-ca\") pod \"88085931-e03b-4e83-a890-e9b7dfafeda1\" (UID: \"88085931-e03b-4e83-a890-e9b7dfafeda1\") "
Aug 02 09:34:16.530263 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:16.530089    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/88085931-e03b-4e83-a890-e9b7dfafeda1-registry-tls\") pod \"88085931-e03b-4e83-a890-e9b7dfafeda1\" (UID: \"88085931-e03b-4e83-a890-e9b7dfafeda1\") "
Aug 02 09:34:16.530263 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:16.530117    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kc75s\" (UniqueName: \"kubernetes.io/projected/88085931-e03b-4e83-a890-e9b7dfafeda1-kube-api-access-kc75s\") pod \"88085931-e03b-4e83-a890-e9b7dfafeda1\" (UID: \"88085931-e03b-4e83-a890-e9b7dfafeda1\") "
Aug 02 09:34:16.530263 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:16.530174    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/88085931-e03b-4e83-a890-e9b7dfafeda1-image-registry-private-configuration\") pod \"88085931-e03b-4e83-a890-e9b7dfafeda1\" (UID: \"88085931-e03b-4e83-a890-e9b7dfafeda1\") "
Aug 02 09:34:16.530263 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:16.530208    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/88085931-e03b-4e83-a890-e9b7dfafeda1-registry-certificates\") pod \"88085931-e03b-4e83-a890-e9b7dfafeda1\" (UID: \"88085931-e03b-4e83-a890-e9b7dfafeda1\") "
Aug 02 09:34:16.530683 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:16.530643    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88085931-e03b-4e83-a890-e9b7dfafeda1-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "88085931-e03b-4e83-a890-e9b7dfafeda1" (UID: "88085931-e03b-4e83-a890-e9b7dfafeda1"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 02 09:34:16.530770 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:16.530689    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88085931-e03b-4e83-a890-e9b7dfafeda1-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "88085931-e03b-4e83-a890-e9b7dfafeda1" (UID: "88085931-e03b-4e83-a890-e9b7dfafeda1"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 02 09:34:16.532579 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:16.532547    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88085931-e03b-4e83-a890-e9b7dfafeda1-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "88085931-e03b-4e83-a890-e9b7dfafeda1" (UID: "88085931-e03b-4e83-a890-e9b7dfafeda1"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 02 09:34:16.532788 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:16.532760    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88085931-e03b-4e83-a890-e9b7dfafeda1-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "88085931-e03b-4e83-a890-e9b7dfafeda1" (UID: "88085931-e03b-4e83-a890-e9b7dfafeda1"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 02 09:34:16.532910 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:16.532844    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88085931-e03b-4e83-a890-e9b7dfafeda1-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "88085931-e03b-4e83-a890-e9b7dfafeda1" (UID: "88085931-e03b-4e83-a890-e9b7dfafeda1"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 02 09:34:16.533041 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:16.533016    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88085931-e03b-4e83-a890-e9b7dfafeda1-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "88085931-e03b-4e83-a890-e9b7dfafeda1" (UID: "88085931-e03b-4e83-a890-e9b7dfafeda1"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 02 09:34:16.533129 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:16.533110    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88085931-e03b-4e83-a890-e9b7dfafeda1-kube-api-access-kc75s" (OuterVolumeSpecName: "kube-api-access-kc75s") pod "88085931-e03b-4e83-a890-e9b7dfafeda1" (UID: "88085931-e03b-4e83-a890-e9b7dfafeda1"). InnerVolumeSpecName "kube-api-access-kc75s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 02 09:34:16.538963 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:16.538934    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88085931-e03b-4e83-a890-e9b7dfafeda1-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "88085931-e03b-4e83-a890-e9b7dfafeda1" (UID: "88085931-e03b-4e83-a890-e9b7dfafeda1"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 02 09:34:16.630803 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:16.630766    2564 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/88085931-e03b-4e83-a890-e9b7dfafeda1-registry-certificates\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 09:34:16.630803 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:16.630799    2564 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/88085931-e03b-4e83-a890-e9b7dfafeda1-installation-pull-secrets\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 09:34:16.630803 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:16.630809    2564 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/88085931-e03b-4e83-a890-e9b7dfafeda1-bound-sa-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 09:34:16.630979 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:16.630818    2564 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/88085931-e03b-4e83-a890-e9b7dfafeda1-ca-trust-extracted\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 09:34:16.630979 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:16.630827    2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/88085931-e03b-4e83-a890-e9b7dfafeda1-trusted-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 09:34:16.630979 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:16.630835    2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/88085931-e03b-4e83-a890-e9b7dfafeda1-registry-tls\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 09:34:16.630979 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:16.630842    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kc75s\" (UniqueName: \"kubernetes.io/projected/88085931-e03b-4e83-a890-e9b7dfafeda1-kube-api-access-kc75s\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 09:34:16.630979 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:16.630851    2564 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/88085931-e03b-4e83-a890-e9b7dfafeda1-image-registry-private-configuration\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 09:34:17.337126 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:17.337028    2564 generic.go:356] "Generic (PLEG): container finished" podID="88085931-e03b-4e83-a890-e9b7dfafeda1" containerID="f1810bca226cdbf7c7e37b9d5cb1adbd896f5256f93294ae34ef3339b0804a6f" exitCode=0
Aug 02 09:34:17.337126 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:17.337082    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f6b7788d4-hl9g6"
Aug 02 09:34:17.337606 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:17.337120    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f6b7788d4-hl9g6" event={"ID":"88085931-e03b-4e83-a890-e9b7dfafeda1","Type":"ContainerDied","Data":"f1810bca226cdbf7c7e37b9d5cb1adbd896f5256f93294ae34ef3339b0804a6f"}
Aug 02 09:34:17.337606 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:17.337192    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f6b7788d4-hl9g6" event={"ID":"88085931-e03b-4e83-a890-e9b7dfafeda1","Type":"ContainerDied","Data":"47b60008d63aa32c9ce74e6cf6561ec96d1cf8ce308da08c443f4d96ce185f20"}
Aug 02 09:34:17.337606 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:17.337223    2564 scope.go:117] "RemoveContainer" containerID="f1810bca226cdbf7c7e37b9d5cb1adbd896f5256f93294ae34ef3339b0804a6f"
Aug 02 09:34:17.347276 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:17.347257    2564 scope.go:117] "RemoveContainer" containerID="f1810bca226cdbf7c7e37b9d5cb1adbd896f5256f93294ae34ef3339b0804a6f"
Aug 02 09:34:17.347589 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:34:17.347566    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1810bca226cdbf7c7e37b9d5cb1adbd896f5256f93294ae34ef3339b0804a6f\": container with ID starting with f1810bca226cdbf7c7e37b9d5cb1adbd896f5256f93294ae34ef3339b0804a6f not found: ID does not exist" containerID="f1810bca226cdbf7c7e37b9d5cb1adbd896f5256f93294ae34ef3339b0804a6f"
Aug 02 09:34:17.347646 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:17.347598    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1810bca226cdbf7c7e37b9d5cb1adbd896f5256f93294ae34ef3339b0804a6f"} err="failed to get container status \"f1810bca226cdbf7c7e37b9d5cb1adbd896f5256f93294ae34ef3339b0804a6f\": rpc error: code = NotFound desc = could not find container \"f1810bca226cdbf7c7e37b9d5cb1adbd896f5256f93294ae34ef3339b0804a6f\": container with ID starting with f1810bca226cdbf7c7e37b9d5cb1adbd896f5256f93294ae34ef3339b0804a6f not found: ID does not exist"
Aug 02 09:34:17.356541 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:17.356515    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5f6b7788d4-hl9g6"]
Aug 02 09:34:17.359003 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:17.358980    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5f6b7788d4-hl9g6"]
Aug 02 09:34:18.266160 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:18.266126    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:18.365465 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:18.365437    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:18.879304 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:18.879264    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88085931-e03b-4e83-a890-e9b7dfafeda1" path="/var/lib/kubelet/pods/88085931-e03b-4e83-a890-e9b7dfafeda1/volumes"
Aug 02 09:34:19.787590 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:19.787555    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:19.888686 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:19.888658    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:20.184973 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:20.184946    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:20.279112 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:20.279081    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:20.369212 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:20.369182    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:20.430043 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:20.430016    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:20.531103 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:20.531039    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:20.847186 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:20.847107    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:20.947600 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:20.947565    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:21.257468 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:21.257436    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:21.357686 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:21.357658    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:21.477839 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:21.477811    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:21.527821 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:21.527767    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:21.857857 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:21.857788    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:22.057759 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:22.057731    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:22.593677 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:22.593643    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:23.678643 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:23.678606    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:23.775369 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:23.775331    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:25.124701 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:25.124669    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:25.225002 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:25.224976    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:28.810705 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:28.810653    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:28.866026 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:28.865998    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:29.635463 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:29.635430    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:32.279933 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:32.279892    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:32.330399 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:32.330370    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:33.271338 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:33.271298    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:33.370508 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:33.370477    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:35.876531 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:35.876498    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:39.749929 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:39.749898    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:39.799884 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:39.799852    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:40.316687 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:40.316660    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-m9jmd"
Aug 02 09:34:43.543695 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:43.543661    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:43.595775 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:43.595743    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:48.266884 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:48.266852    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:48.367885 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:48.367858    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:50.889021 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:50.888992    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:50.938502 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:50.938469    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:52.084566 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:52.084533    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:52.484294 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:52.484228    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:52.534562 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:52.534519    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:34:53.342309 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:53.342285    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-88xrg_b8b72c7d-38e1-4478-ab62-465e91cf5434/init-textfile/0.log"
Aug 02 09:34:53.543263 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:53.543218    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-88xrg_b8b72c7d-38e1-4478-ab62-465e91cf5434/node-exporter/0.log"
Aug 02 09:34:53.742475 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:34:53.742446    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-88xrg_b8b72c7d-38e1-4478-ab62-465e91cf5434/kube-rbac-proxy/0.log"
Aug 02 09:35:03.272251 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:35:03.272198    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:35:03.370125 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:35:03.370097    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:35:23.394433 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:35:23.394399    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:35:23.608268 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:35:23.608180    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:35:31.893761 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:35:31.893728    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:35:31.994075 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:35:31.994047    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:35:32.095333 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:35:32.095304    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:35:32.633349 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:35:32.633318    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:35:32.720082 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:35:32.720057    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:35:43.026975 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:35:43.026939    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:35:43.080185 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:35:43.080162    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:36:20.412534 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:36:20.412452    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:36:20.461811 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:36:20.461771    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:36:38.605008 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:36:38.604975    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:36:38.657172 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:36:38.657147    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:36:50.560597 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:36:50.560560    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:36:50.608952 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:36:50.608925    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:36:53.124606 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:36:53.124570    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:36:53.276931 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:36:53.276894    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:36:53.333936 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:36:53.333905    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:36:53.867207 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:36:53.867172    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:37:03.277604 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:37:03.277554    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:37:03.376763 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:37:03.376738    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:37:04.318614 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:37:04.318579    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:37:04.419948 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:37:04.419917    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:37:04.471286 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:37:04.471257    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:37:20.935394 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:37:20.935304    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:37:45.900567 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:37:45.900529    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:37:46.000277 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:37:46.000229    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:37:56.706619 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:37:56.706586    2564 kubelet.go:1648] "Image garbage collection succeeded"
Aug 02 09:38:07.500615 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:07.500576    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:38:08.648585 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:08.648549    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:38:11.112488 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:11.112450    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:38:11.165059 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:11.165034    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:38:12.769935 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:12.769898    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:38:12.871719 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:12.871687    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:38:13.315938 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:13.315906    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:38:13.393353 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:13.393329    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:38:14.760486 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:14.760455    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:38:14.860761 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:14.860733    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:38:15.325565 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:15.325529    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:38:15.718367 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:15.718334    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:38:17.939606 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:17.939572    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:38:18.031202 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:18.031178    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:38:23.934280 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:23.934248    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:38:26.848561 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:26.848527    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:38:26.948304 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:26.948274    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:38:35.320720 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:35.320688    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:38:35.419427 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:35.419395    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:38:41.439995 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:41.439958    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:38:41.536673 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:41.536643    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:38:43.514581 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:43.514548    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:38:53.754476 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:53.754394    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:38:53.872557 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:53.872529    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-df7456b48-kdms8"]
Aug 02 09:38:53.872769 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:53.872756    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88085931-e03b-4e83-a890-e9b7dfafeda1" containerName="registry"
Aug 02 09:38:53.872816 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:53.872771    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="88085931-e03b-4e83-a890-e9b7dfafeda1" containerName="registry"
Aug 02 09:38:53.872850 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:53.872818    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="88085931-e03b-4e83-a890-e9b7dfafeda1" containerName="registry"
Aug 02 09:38:53.875496 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:53.875481    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-df7456b48-kdms8"
Aug 02 09:38:53.877895 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:53.877875    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Aug 02 09:38:53.878000 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:53.877903    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Aug 02 09:38:53.879002 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:53.878980    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Aug 02 09:38:53.879144 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:53.879016    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Aug 02 09:38:53.879144 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:53.879102    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Aug 02 09:38:53.879263 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:53.879210    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-tsqrk\""
Aug 02 09:38:53.880823 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:53.880800    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:38:53.887514 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:53.887494    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-df7456b48-kdms8"]
Aug 02 09:38:53.926143 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:53.926120    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgv7m\" (UniqueName: \"kubernetes.io/projected/678d742f-961f-4c9a-8a20-bdd3e807bbd6-kube-api-access-kgv7m\") pod \"lws-controller-manager-df7456b48-kdms8\" (UID: \"678d742f-961f-4c9a-8a20-bdd3e807bbd6\") " pod="openshift-lws-operator/lws-controller-manager-df7456b48-kdms8"
Aug 02 09:38:53.926227 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:53.926152    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/678d742f-961f-4c9a-8a20-bdd3e807bbd6-manager-config\") pod \"lws-controller-manager-df7456b48-kdms8\" (UID: \"678d742f-961f-4c9a-8a20-bdd3e807bbd6\") " pod="openshift-lws-operator/lws-controller-manager-df7456b48-kdms8"
Aug 02 09:38:53.926227 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:53.926169    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/678d742f-961f-4c9a-8a20-bdd3e807bbd6-metrics-cert\") pod \"lws-controller-manager-df7456b48-kdms8\" (UID: \"678d742f-961f-4c9a-8a20-bdd3e807bbd6\") " pod="openshift-lws-operator/lws-controller-manager-df7456b48-kdms8"
Aug 02 09:38:53.926331 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:53.926255    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/678d742f-961f-4c9a-8a20-bdd3e807bbd6-cert\") pod \"lws-controller-manager-df7456b48-kdms8\" (UID: \"678d742f-961f-4c9a-8a20-bdd3e807bbd6\") " pod="openshift-lws-operator/lws-controller-manager-df7456b48-kdms8"
Aug 02 09:38:54.027036 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:54.026973    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/678d742f-961f-4c9a-8a20-bdd3e807bbd6-cert\") pod \"lws-controller-manager-df7456b48-kdms8\" (UID: \"678d742f-961f-4c9a-8a20-bdd3e807bbd6\") " pod="openshift-lws-operator/lws-controller-manager-df7456b48-kdms8"
Aug 02 09:38:54.027036 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:54.027026    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgv7m\" (UniqueName: \"kubernetes.io/projected/678d742f-961f-4c9a-8a20-bdd3e807bbd6-kube-api-access-kgv7m\") pod \"lws-controller-manager-df7456b48-kdms8\" (UID: \"678d742f-961f-4c9a-8a20-bdd3e807bbd6\") " pod="openshift-lws-operator/lws-controller-manager-df7456b48-kdms8"
Aug 02 09:38:54.027166 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:54.027053    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/678d742f-961f-4c9a-8a20-bdd3e807bbd6-manager-config\") pod \"lws-controller-manager-df7456b48-kdms8\" (UID: \"678d742f-961f-4c9a-8a20-bdd3e807bbd6\") " pod="openshift-lws-operator/lws-controller-manager-df7456b48-kdms8"
Aug 02 09:38:54.027166 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:54.027080    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/678d742f-961f-4c9a-8a20-bdd3e807bbd6-metrics-cert\") pod \"lws-controller-manager-df7456b48-kdms8\" (UID: \"678d742f-961f-4c9a-8a20-bdd3e807bbd6\") " pod="openshift-lws-operator/lws-controller-manager-df7456b48-kdms8"
Aug 02 09:38:54.027673 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:54.027648    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/678d742f-961f-4c9a-8a20-bdd3e807bbd6-manager-config\") pod \"lws-controller-manager-df7456b48-kdms8\" (UID: \"678d742f-961f-4c9a-8a20-bdd3e807bbd6\") " pod="openshift-lws-operator/lws-controller-manager-df7456b48-kdms8"
Aug 02 09:38:54.029414 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:54.029396    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/678d742f-961f-4c9a-8a20-bdd3e807bbd6-metrics-cert\") pod \"lws-controller-manager-df7456b48-kdms8\" (UID: \"678d742f-961f-4c9a-8a20-bdd3e807bbd6\") " pod="openshift-lws-operator/lws-controller-manager-df7456b48-kdms8"
Aug 02 09:38:54.029530 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:54.029510    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/678d742f-961f-4c9a-8a20-bdd3e807bbd6-cert\") pod \"lws-controller-manager-df7456b48-kdms8\" (UID: \"678d742f-961f-4c9a-8a20-bdd3e807bbd6\") " pod="openshift-lws-operator/lws-controller-manager-df7456b48-kdms8"
Aug 02 09:38:54.041374 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:54.041354    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgv7m\" (UniqueName: \"kubernetes.io/projected/678d742f-961f-4c9a-8a20-bdd3e807bbd6-kube-api-access-kgv7m\") pod \"lws-controller-manager-df7456b48-kdms8\" (UID: \"678d742f-961f-4c9a-8a20-bdd3e807bbd6\") " pod="openshift-lws-operator/lws-controller-manager-df7456b48-kdms8"
Aug 02 09:38:54.184331 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:54.184306    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-df7456b48-kdms8"
Aug 02 09:38:54.304686 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:54.304624    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-df7456b48-kdms8"]
Aug 02 09:38:54.307845 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:38:54.307817    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod678d742f_961f_4c9a_8a20_bdd3e807bbd6.slice/crio-89a97f81e9972b373d184fc2fc73809fc4a5911f7327b957d80b779ad2bb0319 WatchSource:0}: Error finding container 89a97f81e9972b373d184fc2fc73809fc4a5911f7327b957d80b779ad2bb0319: Status 404 returned error can't find the container with id 89a97f81e9972b373d184fc2fc73809fc4a5911f7327b957d80b779ad2bb0319
Aug 02 09:38:54.309680 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:54.309664    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 02 09:38:55.207436 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:55.207306    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-df7456b48-kdms8" event={"ID":"678d742f-961f-4c9a-8a20-bdd3e807bbd6","Type":"ContainerStarted","Data":"89a97f81e9972b373d184fc2fc73809fc4a5911f7327b957d80b779ad2bb0319"}
Aug 02 09:38:55.332570 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:55.332486    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:38:55.425550 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:55.425515    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:38:55.940815 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:55.940766    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:38:56.151710 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:56.151677    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:38:56.253533 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:56.253435    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:38:56.293226 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:56.293197    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5694fb54d4-6z7v6"]
Aug 02 09:38:56.304794 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:56.304765    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5694fb54d4-6z7v6"
Aug 02 09:38:56.310317 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:56.310149    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-m9s7z\""
Aug 02 09:38:56.310317 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:56.310266    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Aug 02 09:38:56.310317 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:56.310269    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Aug 02 09:38:56.310596 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:56.310416    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Aug 02 09:38:56.311218 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:56.311197    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Aug 02 09:38:56.311546 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:56.311525    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5694fb54d4-6z7v6"]
Aug 02 09:38:56.344126 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:56.344103    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/91e8b861-99fe-4311-9db5-ab8763266d31-manifests\") pod \"opendatahub-operator-controller-manager-5694fb54d4-6z7v6\" (UID: \"91e8b861-99fe-4311-9db5-ab8763266d31\") " pod="opendatahub/opendatahub-operator-controller-manager-5694fb54d4-6z7v6"
Aug 02 09:38:56.344262 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:56.344134    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwfh4\" (UniqueName: \"kubernetes.io/projected/91e8b861-99fe-4311-9db5-ab8763266d31-kube-api-access-xwfh4\") pod \"opendatahub-operator-controller-manager-5694fb54d4-6z7v6\" (UID: \"91e8b861-99fe-4311-9db5-ab8763266d31\") " pod="opendatahub/opendatahub-operator-controller-manager-5694fb54d4-6z7v6"
Aug 02 09:38:56.344262 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:56.344163    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/91e8b861-99fe-4311-9db5-ab8763266d31-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5694fb54d4-6z7v6\" (UID: \"91e8b861-99fe-4311-9db5-ab8763266d31\") " pod="opendatahub/opendatahub-operator-controller-manager-5694fb54d4-6z7v6"
Aug 02 09:38:56.344339 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:56.344254    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/91e8b861-99fe-4311-9db5-ab8763266d31-webhook-cert\") pod \"opendatahub-operator-controller-manager-5694fb54d4-6z7v6\" (UID: \"91e8b861-99fe-4311-9db5-ab8763266d31\") " pod="opendatahub/opendatahub-operator-controller-manager-5694fb54d4-6z7v6"
Aug 02 09:38:56.344374 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:56.344340    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91e8b861-99fe-4311-9db5-ab8763266d31-tmp\") pod \"opendatahub-operator-controller-manager-5694fb54d4-6z7v6\" (UID: \"91e8b861-99fe-4311-9db5-ab8763266d31\") " pod="opendatahub/opendatahub-operator-controller-manager-5694fb54d4-6z7v6"
Aug 02 09:38:56.445598 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:56.445562    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/91e8b861-99fe-4311-9db5-ab8763266d31-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5694fb54d4-6z7v6\" (UID: \"91e8b861-99fe-4311-9db5-ab8763266d31\") " pod="opendatahub/opendatahub-operator-controller-manager-5694fb54d4-6z7v6"
Aug 02 09:38:56.445776 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:56.445615    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/91e8b861-99fe-4311-9db5-ab8763266d31-webhook-cert\") pod \"opendatahub-operator-controller-manager-5694fb54d4-6z7v6\" (UID: \"91e8b861-99fe-4311-9db5-ab8763266d31\") " pod="opendatahub/opendatahub-operator-controller-manager-5694fb54d4-6z7v6"
Aug 02 09:38:56.445776 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:56.445651    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91e8b861-99fe-4311-9db5-ab8763266d31-tmp\") pod \"opendatahub-operator-controller-manager-5694fb54d4-6z7v6\" (UID: \"91e8b861-99fe-4311-9db5-ab8763266d31\") " pod="opendatahub/opendatahub-operator-controller-manager-5694fb54d4-6z7v6"
Aug 02 09:38:56.445776 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:56.445699    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/91e8b861-99fe-4311-9db5-ab8763266d31-manifests\") pod \"opendatahub-operator-controller-manager-5694fb54d4-6z7v6\" (UID: \"91e8b861-99fe-4311-9db5-ab8763266d31\") " pod="opendatahub/opendatahub-operator-controller-manager-5694fb54d4-6z7v6"
Aug 02 09:38:56.445776 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:56.445730    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwfh4\" (UniqueName: \"kubernetes.io/projected/91e8b861-99fe-4311-9db5-ab8763266d31-kube-api-access-xwfh4\") pod \"opendatahub-operator-controller-manager-5694fb54d4-6z7v6\" (UID: \"91e8b861-99fe-4311-9db5-ab8763266d31\") " pod="opendatahub/opendatahub-operator-controller-manager-5694fb54d4-6z7v6"
Aug 02 09:38:56.446187 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:56.446155    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/91e8b861-99fe-4311-9db5-ab8763266d31-manifests\") pod \"opendatahub-operator-controller-manager-5694fb54d4-6z7v6\" (UID: \"91e8b861-99fe-4311-9db5-ab8763266d31\") " pod="opendatahub/opendatahub-operator-controller-manager-5694fb54d4-6z7v6"
Aug 02 09:38:56.448086 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:56.448055    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91e8b861-99fe-4311-9db5-ab8763266d31-tmp\") pod \"opendatahub-operator-controller-manager-5694fb54d4-6z7v6\" (UID: \"91e8b861-99fe-4311-9db5-ab8763266d31\") " pod="opendatahub/opendatahub-operator-controller-manager-5694fb54d4-6z7v6"
Aug 02 09:38:56.448337 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:56.448320    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/91e8b861-99fe-4311-9db5-ab8763266d31-webhook-cert\") pod \"opendatahub-operator-controller-manager-5694fb54d4-6z7v6\" (UID: \"91e8b861-99fe-4311-9db5-ab8763266d31\") " pod="opendatahub/opendatahub-operator-controller-manager-5694fb54d4-6z7v6"
Aug 02 09:38:56.448446 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:56.448424    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/91e8b861-99fe-4311-9db5-ab8763266d31-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5694fb54d4-6z7v6\" (UID: \"91e8b861-99fe-4311-9db5-ab8763266d31\") " pod="opendatahub/opendatahub-operator-controller-manager-5694fb54d4-6z7v6"
Aug 02 09:38:56.453984 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:56.453957    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwfh4\" (UniqueName: \"kubernetes.io/projected/91e8b861-99fe-4311-9db5-ab8763266d31-kube-api-access-xwfh4\") pod \"opendatahub-operator-controller-manager-5694fb54d4-6z7v6\" (UID: \"91e8b861-99fe-4311-9db5-ab8763266d31\") " pod="opendatahub/opendatahub-operator-controller-manager-5694fb54d4-6z7v6"
Aug 02 09:38:56.457213 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:56.457190    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:38:56.561703 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:56.561615    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:38:56.618029 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:56.618003    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5694fb54d4-6z7v6"
Aug 02 09:38:56.657926 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:56.657877    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:38:56.743803 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:56.743775    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:38:56.817436 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:56.817409    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:38:56.825202 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:56.825180    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5694fb54d4-6z7v6"]
Aug 02 09:38:56.828284 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:38:56.828257    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e8b861_99fe_4311_9db5_ab8763266d31.slice/crio-71c338e14297d49f784eb0c6b2202139d2efed9aaad4ac0875d52f422e487920 WatchSource:0}: Error finding container 71c338e14297d49f784eb0c6b2202139d2efed9aaad4ac0875d52f422e487920: Status 404 returned error can't find the container with id 71c338e14297d49f784eb0c6b2202139d2efed9aaad4ac0875d52f422e487920
Aug 02 09:38:56.918772 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:56.918736    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:38:57.018612 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:57.018583    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:38:57.215187 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:57.215150    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5694fb54d4-6z7v6" event={"ID":"91e8b861-99fe-4311-9db5-ab8763266d31","Type":"ContainerStarted","Data":"71c338e14297d49f784eb0c6b2202139d2efed9aaad4ac0875d52f422e487920"}
Aug 02 09:38:57.216416 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:57.216392    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-df7456b48-kdms8" event={"ID":"678d742f-961f-4c9a-8a20-bdd3e807bbd6","Type":"ContainerStarted","Data":"a829237f41db34af82844a42a9d31845fe1dba902e257b6e1cb1135be38d5e19"}
Aug 02 09:38:57.216520 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:57.216505    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-df7456b48-kdms8"
Aug 02 09:38:57.232003 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:38:57.231954    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-df7456b48-kdms8" podStartSLOduration=1.781441472 podStartE2EDuration="4.231940157s" totalImagesPullingTime="2.450498685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 09:38:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:38:57.231189872 +0000 UTC m=+360.945995320" watchObservedRunningTime="2026-08-02 09:38:57.231940157 +0000 UTC m=+360.946745604"
Aug 02 09:39:00.229499 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:00.229463    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5694fb54d4-6z7v6" event={"ID":"91e8b861-99fe-4311-9db5-ab8763266d31","Type":"ContainerStarted","Data":"a287358c8a0a21b1cb67c3fb046e64938256937bf51a428485f985703eb713b1"}
Aug 02 09:39:01.233580 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:01.233544    2564 generic.go:356] "Generic (PLEG): container finished" podID="91e8b861-99fe-4311-9db5-ab8763266d31" containerID="a287358c8a0a21b1cb67c3fb046e64938256937bf51a428485f985703eb713b1" exitCode=0
Aug 02 09:39:01.233961 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:01.233600    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5694fb54d4-6z7v6" event={"ID":"91e8b861-99fe-4311-9db5-ab8763266d31","Type":"ContainerDied","Data":"a287358c8a0a21b1cb67c3fb046e64938256937bf51a428485f985703eb713b1"}
Aug 02 09:39:02.239691 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:02.239656    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5694fb54d4-6z7v6" event={"ID":"91e8b861-99fe-4311-9db5-ab8763266d31","Type":"ContainerStarted","Data":"200dda6c665bf47b4c684ec95e17d61b1866160c2730bdbed6a833ae7b22f569"}
Aug 02 09:39:02.240051 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:02.239799    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5694fb54d4-6z7v6"
Aug 02 09:39:02.259038 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:02.258999    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5694fb54d4-6z7v6" podStartSLOduration=2.107861069 podStartE2EDuration="6.25898608s" totalImagesPullingTime="3.142815852s" totalInitContainerRuntime="1.008309159s" isStatefulPod=false podCreationTimestamp="2026-08-02 09:38:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:39:02.257972661 +0000 UTC m=+365.972778108" watchObservedRunningTime="2026-08-02 09:39:02.25898608 +0000 UTC m=+365.973791604"
Aug 02 09:39:08.222522 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:08.222471    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-df7456b48-kdms8"
Aug 02 09:39:09.444405 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:09.444355    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:39:09.544168 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:09.544132    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:39:09.861284 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:09.861177    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:39:09.964709 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:09.964672    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:39:13.245924 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:13.245894    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5694fb54d4-6z7v6"
Aug 02 09:39:15.222178 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:15.222145    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:39:15.329826 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:15.329781    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:39:15.650770 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:15.650732    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:39:15.820879 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:15.820853    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:39:16.086577 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:16.086488    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:39:16.182064 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:16.182037    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:39:16.308934 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:16.308897    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:39:16.388306 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:16.388275    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:39:16.737842 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:16.737770    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:39:16.987502 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:16.987475    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:39:17.054478 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:17.054398    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:39:17.875342 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:17.875310    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:39:22.777524 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:22.777489    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-cbzmt"]
Aug 02 09:39:22.780402 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:22.780375    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cbzmt"
Aug 02 09:39:22.783579 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:22.783558    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Aug 02 09:39:22.783700 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:22.783571    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Aug 02 09:39:22.783824 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:22.783808    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Aug 02 09:39:22.784494 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:22.784479    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Aug 02 09:39:22.784651 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:22.784635    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Aug 02 09:39:22.785875 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:22.785857    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Aug 02 09:39:22.786444 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:22.786421    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-844d4\""
Aug 02 09:39:22.794523 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:22.794493    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-cbzmt"]
Aug 02 09:39:22.822272 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:22.822222    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/84d87887-e1dd-4a84-807d-6f551f7cbe8a-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-cbzmt\" (UID: \"84d87887-e1dd-4a84-807d-6f551f7cbe8a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cbzmt"
Aug 02 09:39:22.822381 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:22.822284    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84d87887-e1dd-4a84-807d-6f551f7cbe8a-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-cbzmt\" (UID: \"84d87887-e1dd-4a84-807d-6f551f7cbe8a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cbzmt"
Aug 02 09:39:22.822381 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:22.822310    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/84d87887-e1dd-4a84-807d-6f551f7cbe8a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-cbzmt\" (UID: \"84d87887-e1dd-4a84-807d-6f551f7cbe8a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cbzmt"
Aug 02 09:39:22.822381 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:22.822376    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/84d87887-e1dd-4a84-807d-6f551f7cbe8a-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-cbzmt\" (UID: \"84d87887-e1dd-4a84-807d-6f551f7cbe8a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cbzmt"
Aug 02 09:39:22.822535 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:22.822416    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/84d87887-e1dd-4a84-807d-6f551f7cbe8a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-cbzmt\" (UID: \"84d87887-e1dd-4a84-807d-6f551f7cbe8a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cbzmt"
Aug 02 09:39:22.822535 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:22.822458    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/84d87887-e1dd-4a84-807d-6f551f7cbe8a-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-cbzmt\" (UID: \"84d87887-e1dd-4a84-807d-6f551f7cbe8a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cbzmt"
Aug 02 09:39:22.822535 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:22.822486    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndg78\" (UniqueName: \"kubernetes.io/projected/84d87887-e1dd-4a84-807d-6f551f7cbe8a-kube-api-access-ndg78\") pod \"istiod-openshift-gateway-5677ccb49b-cbzmt\" (UID: \"84d87887-e1dd-4a84-807d-6f551f7cbe8a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cbzmt"
Aug 02 09:39:22.923822 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:22.923793    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/84d87887-e1dd-4a84-807d-6f551f7cbe8a-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-cbzmt\" (UID: \"84d87887-e1dd-4a84-807d-6f551f7cbe8a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cbzmt"
Aug 02 09:39:22.923822 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:22.923823    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84d87887-e1dd-4a84-807d-6f551f7cbe8a-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-cbzmt\" (UID: \"84d87887-e1dd-4a84-807d-6f551f7cbe8a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cbzmt"
Aug 02 09:39:22.924022 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:22.923841    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/84d87887-e1dd-4a84-807d-6f551f7cbe8a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-cbzmt\" (UID: \"84d87887-e1dd-4a84-807d-6f551f7cbe8a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cbzmt"
Aug 02 09:39:22.924022 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:22.923989    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/84d87887-e1dd-4a84-807d-6f551f7cbe8a-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-cbzmt\" (UID: \"84d87887-e1dd-4a84-807d-6f551f7cbe8a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cbzmt"
Aug 02 09:39:22.924135 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:22.924030    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/84d87887-e1dd-4a84-807d-6f551f7cbe8a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-cbzmt\" (UID: \"84d87887-e1dd-4a84-807d-6f551f7cbe8a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cbzmt"
Aug 02 09:39:22.924135 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:22.924077    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/84d87887-e1dd-4a84-807d-6f551f7cbe8a-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-cbzmt\" (UID: \"84d87887-e1dd-4a84-807d-6f551f7cbe8a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cbzmt"
Aug 02 09:39:22.924135 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:22.924103    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndg78\" (UniqueName: \"kubernetes.io/projected/84d87887-e1dd-4a84-807d-6f551f7cbe8a-kube-api-access-ndg78\") pod \"istiod-openshift-gateway-5677ccb49b-cbzmt\" (UID: \"84d87887-e1dd-4a84-807d-6f551f7cbe8a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cbzmt"
Aug 02 09:39:22.924717 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:22.924693    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/84d87887-e1dd-4a84-807d-6f551f7cbe8a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-cbzmt\" (UID: \"84d87887-e1dd-4a84-807d-6f551f7cbe8a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cbzmt"
Aug 02 09:39:22.926382 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:22.926359    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/84d87887-e1dd-4a84-807d-6f551f7cbe8a-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-cbzmt\" (UID: \"84d87887-e1dd-4a84-807d-6f551f7cbe8a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cbzmt"
Aug 02 09:39:22.926526 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:22.926498    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/84d87887-e1dd-4a84-807d-6f551f7cbe8a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-cbzmt\" (UID: \"84d87887-e1dd-4a84-807d-6f551f7cbe8a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cbzmt"
Aug 02 09:39:22.926526 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:22.926505    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/84d87887-e1dd-4a84-807d-6f551f7cbe8a-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-cbzmt\" (UID: \"84d87887-e1dd-4a84-807d-6f551f7cbe8a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cbzmt"
Aug 02 09:39:22.926645 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:22.926588    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/84d87887-e1dd-4a84-807d-6f551f7cbe8a-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-cbzmt\" (UID: \"84d87887-e1dd-4a84-807d-6f551f7cbe8a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cbzmt"
Aug 02 09:39:22.930882 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:22.930861    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndg78\" (UniqueName: \"kubernetes.io/projected/84d87887-e1dd-4a84-807d-6f551f7cbe8a-kube-api-access-ndg78\") pod \"istiod-openshift-gateway-5677ccb49b-cbzmt\" (UID: \"84d87887-e1dd-4a84-807d-6f551f7cbe8a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cbzmt"
Aug 02 09:39:22.931272 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:22.931255    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84d87887-e1dd-4a84-807d-6f551f7cbe8a-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-cbzmt\" (UID: \"84d87887-e1dd-4a84-807d-6f551f7cbe8a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cbzmt"
Aug 02 09:39:23.090604 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:23.090528    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cbzmt"
Aug 02 09:39:23.213390 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:23.213360    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:39:23.232334 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:23.232307    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-cbzmt"]
Aug 02 09:39:23.235287 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:39:23.235257    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84d87887_e1dd_4a84_807d_6f551f7cbe8a.slice/crio-501cf9a783f5cd8b36a2b09620330507064938a6778c0c36a9984723568995fc WatchSource:0}: Error finding container 501cf9a783f5cd8b36a2b09620330507064938a6778c0c36a9984723568995fc: Status 404 returned error can't find the container with id 501cf9a783f5cd8b36a2b09620330507064938a6778c0c36a9984723568995fc
Aug 02 09:39:23.309802 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:23.309778    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:39:23.312494 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:23.312473    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cbzmt" event={"ID":"84d87887-e1dd-4a84-807d-6f551f7cbe8a","Type":"ContainerStarted","Data":"501cf9a783f5cd8b36a2b09620330507064938a6778c0c36a9984723568995fc"}
Aug 02 09:39:23.568018 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:23.567991    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:39:23.675460 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:23.675429    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:39:23.776826 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:23.776755    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:39:25.401463 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:25.401430    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:39:25.519929 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:25.519882    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Aug 02 09:39:25.520063 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:25.519967    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Aug 02 09:39:25.540952 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:25.540930    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:39:25.829618 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:25.829536    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:39:25.950118 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:25.950086    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:39:26.325225 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:26.325192    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cbzmt" event={"ID":"84d87887-e1dd-4a84-807d-6f551f7cbe8a","Type":"ContainerStarted","Data":"896e3c7329a3db6690aaf0b3a7036a761f47e0e991528cd8a4d57f428c571c1b"}
Aug 02 09:39:26.325430 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:26.325376    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cbzmt"
Aug 02 09:39:26.326906 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:26.326879    2564 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-cbzmt container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Aug 02 09:39:26.327013 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:26.326943    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cbzmt" podUID="84d87887-e1dd-4a84-807d-6f551f7cbe8a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 02 09:39:26.375356 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:26.375313    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cbzmt" podStartSLOduration=2.093137922 podStartE2EDuration="4.375301769s" totalImagesPullingTime="2.282163847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 09:39:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:39:26.373934506 +0000 UTC m=+390.088739955" watchObservedRunningTime="2026-08-02 09:39:26.375301769 +0000 UTC m=+390.090107217"
Aug 02 09:39:26.693619 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:26.693591    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:39:26.794897 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:26.794868    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:39:26.908882 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:26.908852    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:39:27.329418 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:27.329385    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cbzmt"
Aug 02 09:39:32.270092 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:32.270053    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:39:32.371783 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:32.371752    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:39:32.775392 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:32.775356    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:39:40.554125 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:40.554088    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:39:40.641846 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:40.641816    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:39:44.103165 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:44.103132    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:39:44.151580 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:44.151552    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:39:44.203432 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:39:44.203403    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:40:17.046209 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:40:17.046171    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:40:17.144624 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:40:17.144594    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:40:19.513437 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:40:19.513364    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:40:19.579438 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:40:19.579400    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:40:21.456026 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:40:21.455997    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:40:21.507345 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:40:21.507319    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:40:23.761362 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:40:23.761325    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:40:23.811658 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:40:23.811623    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:40:28.638399 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:40:28.638360    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:40:31.043583 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:40:31.043547    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:40:31.439784 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:40:31.439749    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:40:33.376361 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:40:33.376331    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:40:33.468762 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:40:33.468726    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:40:33.839899 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:40:33.839823    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:40:34.235631 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:40:34.235602    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:40:34.638726 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:40:34.638664    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:40:35.342511 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:40:35.342475    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:40:36.299146 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:40:36.299113    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:40:36.405245 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:40:36.405198    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:40:38.100952 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:40:38.100912    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:40:38.276342 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:40:38.276305    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:40:41.047758 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:40:41.047727    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:40:41.150129 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:40:41.150092    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:40:44.471958 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:40:44.471912    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:40:44.898542 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:40:44.898502    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:40:45.042276 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:40:45.042226    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:40:45.142987 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:40:45.142958    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:40:45.243023 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:40:45.242932    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:40:45.343006 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:40:45.342977    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:40:47.039749 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:40:47.039718    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:40:50.181521 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:40:50.181491    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:40:50.281099 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:40:50.281068    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:40:50.381460 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:40:50.381429    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:40:56.726265 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:40:56.726217    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:40:56.826997 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:40:56.826964    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:41:01.057081 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:01.057040    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:41:01.157687 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:01.157654    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:41:04.821879 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:04.821846    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:41:04.922663 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:04.922622    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:41:05.022288 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:05.022225    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:41:05.082165 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:05.082084    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:41:10.427444 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:10.427416    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:41:10.528356 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:10.528327    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:41:10.630183 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:10.630153    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:41:10.686224 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:10.686141    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:41:10.826928 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:10.826892    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:41:10.928069 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:10.928038    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:41:16.736984 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:16.736953    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:41:16.837959 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:16.837909    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:41:20.167723 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:20.167687    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:41:20.267919 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:20.267888    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:41:23.822488 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:23.822459    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:41:24.416697 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:24.416637    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:41:25.563113 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:25.563057    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-hmfl2"]
Aug 02 09:41:25.563489 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:25.563302    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:41:25.566192 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:25.566177    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-hmfl2"
Aug 02 09:41:25.571185 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:25.571152    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Aug 02 09:41:25.571344 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:25.571153    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Aug 02 09:41:25.571789 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:25.571773    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-rrmvk\""
Aug 02 09:41:25.571870 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:25.571834    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Aug 02 09:41:25.577804 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:25.577781    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-hmfl2"]
Aug 02 09:41:25.660447 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:25.660414    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-hmfl2"]
Aug 02 09:41:25.673544 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:25.673517    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:41:25.725703 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:25.725664    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5615fe70-e0fa-4183-96d2-086a02ae34ce-config-file\") pod \"limitador-limitador-859c4d96d6-hmfl2\" (UID: \"5615fe70-e0fa-4183-96d2-086a02ae34ce\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-hmfl2"
Aug 02 09:41:25.725885 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:25.725751    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z6rw\" (UniqueName: \"kubernetes.io/projected/5615fe70-e0fa-4183-96d2-086a02ae34ce-kube-api-access-4z6rw\") pod \"limitador-limitador-859c4d96d6-hmfl2\" (UID: \"5615fe70-e0fa-4183-96d2-086a02ae34ce\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-hmfl2"
Aug 02 09:41:25.727422 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:25.727386    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:41:25.826969 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:25.826883    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z6rw\" (UniqueName: \"kubernetes.io/projected/5615fe70-e0fa-4183-96d2-086a02ae34ce-kube-api-access-4z6rw\") pod \"limitador-limitador-859c4d96d6-hmfl2\" (UID: \"5615fe70-e0fa-4183-96d2-086a02ae34ce\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-hmfl2"
Aug 02 09:41:25.826969 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:25.826926    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5615fe70-e0fa-4183-96d2-086a02ae34ce-config-file\") pod \"limitador-limitador-859c4d96d6-hmfl2\" (UID: \"5615fe70-e0fa-4183-96d2-086a02ae34ce\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-hmfl2"
Aug 02 09:41:25.827516 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:25.827497    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5615fe70-e0fa-4183-96d2-086a02ae34ce-config-file\") pod \"limitador-limitador-859c4d96d6-hmfl2\" (UID: \"5615fe70-e0fa-4183-96d2-086a02ae34ce\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-hmfl2"
Aug 02 09:41:25.835012 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:25.834987    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z6rw\" (UniqueName: \"kubernetes.io/projected/5615fe70-e0fa-4183-96d2-086a02ae34ce-kube-api-access-4z6rw\") pod \"limitador-limitador-859c4d96d6-hmfl2\" (UID: \"5615fe70-e0fa-4183-96d2-086a02ae34ce\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-hmfl2"
Aug 02 09:41:25.875609 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:25.875577    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-hmfl2"
Aug 02 09:41:25.997148 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:25.997022    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-hmfl2"]
Aug 02 09:41:25.999819 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:41:25.999785    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5615fe70_e0fa_4183_96d2_086a02ae34ce.slice/crio-fe031ecd6a0a598ca9fe5eea3b6c3e093dccdf4b566167cff752a1f65a2f9cc7 WatchSource:0}: Error finding container fe031ecd6a0a598ca9fe5eea3b6c3e093dccdf4b566167cff752a1f65a2f9cc7: Status 404 returned error can't find the container with id fe031ecd6a0a598ca9fe5eea3b6c3e093dccdf4b566167cff752a1f65a2f9cc7
Aug 02 09:41:26.399101 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:26.399059    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:41:26.499070 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:26.499037    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:41:26.741978 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:26.741902    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-hmfl2" event={"ID":"5615fe70-e0fa-4183-96d2-086a02ae34ce","Type":"ContainerStarted","Data":"fe031ecd6a0a598ca9fe5eea3b6c3e093dccdf4b566167cff752a1f65a2f9cc7"}
Aug 02 09:41:28.752312 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:28.752280    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-hmfl2" event={"ID":"5615fe70-e0fa-4183-96d2-086a02ae34ce","Type":"ContainerStarted","Data":"aed811a015c409fb56ad56dbec1c86291d7e3c950875c90297c66b2abeb2a39b"}
Aug 02 09:41:28.752739 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:28.752391    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-hmfl2"
Aug 02 09:41:28.769878 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:28.769789    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-859c4d96d6-hmfl2" podStartSLOduration=1.25288866 podStartE2EDuration="3.76977483s" totalImagesPullingTime="2.51688617s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 09:41:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:41:28.768013071 +0000 UTC m=+512.482818516" watchObservedRunningTime="2026-08-02 09:41:28.76977483 +0000 UTC m=+512.484580279"
Aug 02 09:41:39.756909 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:39.756876    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-hmfl2"
Aug 02 09:41:43.314870 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:43.314840    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-hmfl2"]
Aug 02 09:41:43.315277 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:43.315009    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-859c4d96d6-hmfl2" podUID="5615fe70-e0fa-4183-96d2-086a02ae34ce" containerName="limitador" containerID="cri-o://aed811a015c409fb56ad56dbec1c86291d7e3c950875c90297c66b2abeb2a39b" gracePeriod=30
Aug 02 09:41:43.809447 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:43.809413    2564 generic.go:356] "Generic (PLEG): container finished" podID="5615fe70-e0fa-4183-96d2-086a02ae34ce" containerID="aed811a015c409fb56ad56dbec1c86291d7e3c950875c90297c66b2abeb2a39b" exitCode=0
Aug 02 09:41:43.809591 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:43.809458    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-hmfl2" event={"ID":"5615fe70-e0fa-4183-96d2-086a02ae34ce","Type":"ContainerDied","Data":"aed811a015c409fb56ad56dbec1c86291d7e3c950875c90297c66b2abeb2a39b"}
Aug 02 09:41:44.250451 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:44.250430    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-hmfl2"
Aug 02 09:41:44.255013 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:44.254972    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4z6rw\" (UniqueName: \"kubernetes.io/projected/5615fe70-e0fa-4183-96d2-086a02ae34ce-kube-api-access-4z6rw\") pod \"5615fe70-e0fa-4183-96d2-086a02ae34ce\" (UID: \"5615fe70-e0fa-4183-96d2-086a02ae34ce\") "
Aug 02 09:41:44.255077 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:44.255051    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5615fe70-e0fa-4183-96d2-086a02ae34ce-config-file\") pod \"5615fe70-e0fa-4183-96d2-086a02ae34ce\" (UID: \"5615fe70-e0fa-4183-96d2-086a02ae34ce\") "
Aug 02 09:41:44.255419 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:44.255396    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5615fe70-e0fa-4183-96d2-086a02ae34ce-config-file" (OuterVolumeSpecName: "config-file") pod "5615fe70-e0fa-4183-96d2-086a02ae34ce" (UID: "5615fe70-e0fa-4183-96d2-086a02ae34ce"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 02 09:41:44.256988 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:44.256967    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5615fe70-e0fa-4183-96d2-086a02ae34ce-kube-api-access-4z6rw" (OuterVolumeSpecName: "kube-api-access-4z6rw") pod "5615fe70-e0fa-4183-96d2-086a02ae34ce" (UID: "5615fe70-e0fa-4183-96d2-086a02ae34ce"). InnerVolumeSpecName "kube-api-access-4z6rw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 02 09:41:44.355495 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:44.355470    2564 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5615fe70-e0fa-4183-96d2-086a02ae34ce-config-file\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 09:41:44.355495 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:44.355497    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z6rw\" (UniqueName: \"kubernetes.io/projected/5615fe70-e0fa-4183-96d2-086a02ae34ce-kube-api-access-4z6rw\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 09:41:44.814545 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:44.814505    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-hmfl2" event={"ID":"5615fe70-e0fa-4183-96d2-086a02ae34ce","Type":"ContainerDied","Data":"fe031ecd6a0a598ca9fe5eea3b6c3e093dccdf4b566167cff752a1f65a2f9cc7"}
Aug 02 09:41:44.814545 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:44.814531    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-hmfl2"
Aug 02 09:41:44.814752 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:44.814574    2564 scope.go:117] "RemoveContainer" containerID="aed811a015c409fb56ad56dbec1c86291d7e3c950875c90297c66b2abeb2a39b"
Aug 02 09:41:44.833659 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:44.833635    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-hmfl2"]
Aug 02 09:41:44.835684 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:44.835665    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-hmfl2"]
Aug 02 09:41:44.879438 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:44.879414    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5615fe70-e0fa-4183-96d2-086a02ae34ce" path="/var/lib/kubelet/pods/5615fe70-e0fa-4183-96d2-086a02ae34ce/volumes"
Aug 02 09:41:46.818628 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:46.818591    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:41:46.916136 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:46.916105    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:41:47.274714 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:47.274673    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-2jcgx"]
Aug 02 09:41:47.274971 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:47.274956    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5615fe70-e0fa-4183-96d2-086a02ae34ce" containerName="limitador"
Aug 02 09:41:47.275064 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:47.274974    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5615fe70-e0fa-4183-96d2-086a02ae34ce" containerName="limitador"
Aug 02 09:41:47.275064 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:47.275047    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5615fe70-e0fa-4183-96d2-086a02ae34ce" containerName="limitador"
Aug 02 09:41:47.279330 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:47.279308    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2jcgx"
Aug 02 09:41:47.281424 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:47.281399    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Aug 02 09:41:47.281424 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:47.281416    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-n4dk8\""
Aug 02 09:41:47.282155 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:47.282117    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Aug 02 09:41:47.282287 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:47.282161    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Aug 02 09:41:47.285660 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:47.285640    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-2jcgx"]
Aug 02 09:41:47.308517 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:47.308492    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:41:47.373320 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:47.373300    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e2233455-e64a-41d2-a92d-f884f674d629-data\") pod \"postgres-69fb4594cf-2jcgx\" (UID: \"e2233455-e64a-41d2-a92d-f884f674d629\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2jcgx"
Aug 02 09:41:47.373406 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:47.373353    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b6w4\" (UniqueName: \"kubernetes.io/projected/e2233455-e64a-41d2-a92d-f884f674d629-kube-api-access-7b6w4\") pod \"postgres-69fb4594cf-2jcgx\" (UID: \"e2233455-e64a-41d2-a92d-f884f674d629\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2jcgx"
Aug 02 09:41:47.473663 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:47.473630    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b6w4\" (UniqueName: \"kubernetes.io/projected/e2233455-e64a-41d2-a92d-f884f674d629-kube-api-access-7b6w4\") pod \"postgres-69fb4594cf-2jcgx\" (UID: \"e2233455-e64a-41d2-a92d-f884f674d629\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2jcgx"
Aug 02 09:41:47.473821 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:47.473673    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e2233455-e64a-41d2-a92d-f884f674d629-data\") pod \"postgres-69fb4594cf-2jcgx\" (UID: \"e2233455-e64a-41d2-a92d-f884f674d629\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2jcgx"
Aug 02 09:41:47.474020 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:47.474004    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e2233455-e64a-41d2-a92d-f884f674d629-data\") pod \"postgres-69fb4594cf-2jcgx\" (UID: \"e2233455-e64a-41d2-a92d-f884f674d629\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2jcgx"
Aug 02 09:41:47.481299 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:47.481280    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b6w4\" (UniqueName: \"kubernetes.io/projected/e2233455-e64a-41d2-a92d-f884f674d629-kube-api-access-7b6w4\") pod \"postgres-69fb4594cf-2jcgx\" (UID: \"e2233455-e64a-41d2-a92d-f884f674d629\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2jcgx"
Aug 02 09:41:47.590155 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:47.590088    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2jcgx"
Aug 02 09:41:47.711541 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:47.711513    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-2jcgx"]
Aug 02 09:41:47.713593 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:41:47.713565    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2233455_e64a_41d2_a92d_f884f674d629.slice/crio-e8a14139bfb421895610c8a4715fdbfa27a999ced2db707c31bb7d22643bcb7b WatchSource:0}: Error finding container e8a14139bfb421895610c8a4715fdbfa27a999ced2db707c31bb7d22643bcb7b: Status 404 returned error can't find the container with id e8a14139bfb421895610c8a4715fdbfa27a999ced2db707c31bb7d22643bcb7b
Aug 02 09:41:47.827552 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:47.827522    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2jcgx" event={"ID":"e2233455-e64a-41d2-a92d-f884f674d629","Type":"ContainerStarted","Data":"e8a14139bfb421895610c8a4715fdbfa27a999ced2db707c31bb7d22643bcb7b"}
Aug 02 09:41:52.850266 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:52.850213    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2jcgx" event={"ID":"e2233455-e64a-41d2-a92d-f884f674d629","Type":"ContainerStarted","Data":"7777a6b63a8b7d73369813e621015264dfddaaf747f0b5362710561e7a648f9e"}
Aug 02 09:41:52.850666 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:52.850328    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2jcgx"
Aug 02 09:41:52.865335 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:52.865288    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2jcgx" podStartSLOduration=1.258641128 podStartE2EDuration="5.865274985s" totalImagesPullingTime="4.606633857s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 09:41:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:41:52.864928518 +0000 UTC m=+536.579733967" watchObservedRunningTime="2026-08-02 09:41:52.865274985 +0000 UTC m=+536.580080433"
Aug 02 09:41:58.881522 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:58.881483    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2jcgx"
Aug 02 09:41:59.189274 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:59.189182    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:41:59.366846 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:41:59.366788    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:42:06.184650 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:06.184617    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:42:06.318745 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:06.318705    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:42:06.377566 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:06.377517    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:42:06.480629 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:06.480553    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:42:08.316122 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:08.316092    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:42:08.416620 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:08.416588    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:42:08.674732 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:08.674702    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:42:40.122998 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:40.122960    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:42:40.259010 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:40.258971    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:42:41.574671 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:41.574636    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:42:41.625623 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:41.625589    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-856cd5f99f-rv47z"]
Aug 02 09:42:41.628764 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:41.628747    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-856cd5f99f-rv47z"
Aug 02 09:42:41.631218 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:41.631197    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-swwbg\""
Aug 02 09:42:41.631352 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:41.631262    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Aug 02 09:42:41.650603 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:41.650575    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-856cd5f99f-rv47z"]
Aug 02 09:42:41.780437 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:41.780400    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c674e553-e960-4aa7-8ccf-ef3da49ffa4f-maas-api-tls\") pod \"maas-api-856cd5f99f-rv47z\" (UID: \"c674e553-e960-4aa7-8ccf-ef3da49ffa4f\") " pod="odh-ai-gateway-infra/maas-api-856cd5f99f-rv47z"
Aug 02 09:42:41.780437 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:41.780442    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c674e553-e960-4aa7-8ccf-ef3da49ffa4f-openshift-service-ca\") pod \"maas-api-856cd5f99f-rv47z\" (UID: \"c674e553-e960-4aa7-8ccf-ef3da49ffa4f\") " pod="odh-ai-gateway-infra/maas-api-856cd5f99f-rv47z"
Aug 02 09:42:41.780745 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:41.780511    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-666b5\" (UniqueName: \"kubernetes.io/projected/c674e553-e960-4aa7-8ccf-ef3da49ffa4f-kube-api-access-666b5\") pod \"maas-api-856cd5f99f-rv47z\" (UID: \"c674e553-e960-4aa7-8ccf-ef3da49ffa4f\") " pod="odh-ai-gateway-infra/maas-api-856cd5f99f-rv47z"
Aug 02 09:42:41.881768 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:41.881735    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-666b5\" (UniqueName: \"kubernetes.io/projected/c674e553-e960-4aa7-8ccf-ef3da49ffa4f-kube-api-access-666b5\") pod \"maas-api-856cd5f99f-rv47z\" (UID: \"c674e553-e960-4aa7-8ccf-ef3da49ffa4f\") " pod="odh-ai-gateway-infra/maas-api-856cd5f99f-rv47z"
Aug 02 09:42:41.881957 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:41.881793    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c674e553-e960-4aa7-8ccf-ef3da49ffa4f-maas-api-tls\") pod \"maas-api-856cd5f99f-rv47z\" (UID: \"c674e553-e960-4aa7-8ccf-ef3da49ffa4f\") " pod="odh-ai-gateway-infra/maas-api-856cd5f99f-rv47z"
Aug 02 09:42:41.881957 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:42:41.881933    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found
Aug 02 09:42:41.882073 ip-10-0-141-232 kubenswrapper[2564]: E0802 09:42:41.881978    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c674e553-e960-4aa7-8ccf-ef3da49ffa4f-maas-api-tls podName:c674e553-e960-4aa7-8ccf-ef3da49ffa4f nodeName:}" failed. No retries permitted until 2026-08-02 09:42:42.381963174 +0000 UTC m=+586.096768600 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c674e553-e960-4aa7-8ccf-ef3da49ffa4f-maas-api-tls") pod "maas-api-856cd5f99f-rv47z" (UID: "c674e553-e960-4aa7-8ccf-ef3da49ffa4f") : secret "maas-api-serving-cert" not found
Aug 02 09:42:41.882073 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:41.882008    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c674e553-e960-4aa7-8ccf-ef3da49ffa4f-openshift-service-ca\") pod \"maas-api-856cd5f99f-rv47z\" (UID: \"c674e553-e960-4aa7-8ccf-ef3da49ffa4f\") " pod="odh-ai-gateway-infra/maas-api-856cd5f99f-rv47z"
Aug 02 09:42:41.882605 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:41.882585    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c674e553-e960-4aa7-8ccf-ef3da49ffa4f-openshift-service-ca\") pod \"maas-api-856cd5f99f-rv47z\" (UID: \"c674e553-e960-4aa7-8ccf-ef3da49ffa4f\") " pod="odh-ai-gateway-infra/maas-api-856cd5f99f-rv47z"
Aug 02 09:42:41.893064 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:41.893041    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-666b5\" (UniqueName: \"kubernetes.io/projected/c674e553-e960-4aa7-8ccf-ef3da49ffa4f-kube-api-access-666b5\") pod \"maas-api-856cd5f99f-rv47z\" (UID: \"c674e553-e960-4aa7-8ccf-ef3da49ffa4f\") " pod="odh-ai-gateway-infra/maas-api-856cd5f99f-rv47z"
Aug 02 09:42:42.141073 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:42.140985    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:42:42.362119 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:42.362082    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:42:42.386074 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:42.386043    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c674e553-e960-4aa7-8ccf-ef3da49ffa4f-maas-api-tls\") pod \"maas-api-856cd5f99f-rv47z\" (UID: \"c674e553-e960-4aa7-8ccf-ef3da49ffa4f\") " pod="odh-ai-gateway-infra/maas-api-856cd5f99f-rv47z"
Aug 02 09:42:42.388668 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:42.388639    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c674e553-e960-4aa7-8ccf-ef3da49ffa4f-maas-api-tls\") pod \"maas-api-856cd5f99f-rv47z\" (UID: \"c674e553-e960-4aa7-8ccf-ef3da49ffa4f\") " pod="odh-ai-gateway-infra/maas-api-856cd5f99f-rv47z"
Aug 02 09:42:42.478427 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:42.478337    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:42:42.538510 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:42.538475    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-856cd5f99f-rv47z"
Aug 02 09:42:42.599306 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:42.599264    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-hfw8f"]
Aug 02 09:42:42.604002 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:42.603655    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-hfw8f"
Aug 02 09:42:42.607658 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:42.607634    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Aug 02 09:42:42.634102 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:42.634080    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-hfw8f"]
Aug 02 09:42:42.690142 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:42.690103    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d41871d5-cf2c-4676-8d98-96060bef7b10-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-hfw8f\" (UID: \"d41871d5-cf2c-4676-8d98-96060bef7b10\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-hfw8f"
Aug 02 09:42:42.690142 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:42.690143    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2pz9\" (UniqueName: \"kubernetes.io/projected/d41871d5-cf2c-4676-8d98-96060bef7b10-kube-api-access-n2pz9\") pod \"payload-pre-processing-687dbb6cf6-hfw8f\" (UID: \"d41871d5-cf2c-4676-8d98-96060bef7b10\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-hfw8f"
Aug 02 09:42:42.712076 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:42.712050    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-856cd5f99f-rv47z"]
Aug 02 09:42:42.713094 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:42.713071    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-7b8zc"]
Aug 02 09:42:42.714277 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:42:42.714252    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc674e553_e960_4aa7_8ccf_ef3da49ffa4f.slice/crio-908b1c58bcefcfcd0ea70a8b5593252e84489fef3900a61df2bf1a91b3092ee2 WatchSource:0}: Error finding container 908b1c58bcefcfcd0ea70a8b5593252e84489fef3900a61df2bf1a91b3092ee2: Status 404 returned error can't find the container with id 908b1c58bcefcfcd0ea70a8b5593252e84489fef3900a61df2bf1a91b3092ee2
Aug 02 09:42:42.716159 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:42.716139    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-7b8zc"
Aug 02 09:42:42.729027 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:42.728976    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-zcwxk\""
Aug 02 09:42:42.738606 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:42.738581    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-7b8zc"]
Aug 02 09:42:42.791136 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:42.791100    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d41871d5-cf2c-4676-8d98-96060bef7b10-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-hfw8f\" (UID: \"d41871d5-cf2c-4676-8d98-96060bef7b10\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-hfw8f"
Aug 02 09:42:42.791136 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:42.791139    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2pz9\" (UniqueName: \"kubernetes.io/projected/d41871d5-cf2c-4676-8d98-96060bef7b10-kube-api-access-n2pz9\") pod \"payload-pre-processing-687dbb6cf6-hfw8f\" (UID: \"d41871d5-cf2c-4676-8d98-96060bef7b10\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-hfw8f"
Aug 02 09:42:42.791364 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:42.791191    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eda18ed-b699-4efd-b283-47c0ac424ecb-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-7b8zc\" (UID: \"9eda18ed-b699-4efd-b283-47c0ac424ecb\") " pod="openshift-ingress/payload-processing-5cc6cd444b-7b8zc"
Aug 02 09:42:42.791364 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:42.791248    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5shgw\" (UniqueName: \"kubernetes.io/projected/9eda18ed-b699-4efd-b283-47c0ac424ecb-kube-api-access-5shgw\") pod \"payload-processing-5cc6cd444b-7b8zc\" (UID: \"9eda18ed-b699-4efd-b283-47c0ac424ecb\") " pod="openshift-ingress/payload-processing-5cc6cd444b-7b8zc"
Aug 02 09:42:42.791811 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:42.791793    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d41871d5-cf2c-4676-8d98-96060bef7b10-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-hfw8f\" (UID: \"d41871d5-cf2c-4676-8d98-96060bef7b10\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-hfw8f"
Aug 02 09:42:42.804197 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:42.804166    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2pz9\" (UniqueName: \"kubernetes.io/projected/d41871d5-cf2c-4676-8d98-96060bef7b10-kube-api-access-n2pz9\") pod \"payload-pre-processing-687dbb6cf6-hfw8f\" (UID: \"d41871d5-cf2c-4676-8d98-96060bef7b10\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-hfw8f"
Aug 02 09:42:42.892466 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:42.892427    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5shgw\" (UniqueName: \"kubernetes.io/projected/9eda18ed-b699-4efd-b283-47c0ac424ecb-kube-api-access-5shgw\") pod \"payload-processing-5cc6cd444b-7b8zc\" (UID: \"9eda18ed-b699-4efd-b283-47c0ac424ecb\") " pod="openshift-ingress/payload-processing-5cc6cd444b-7b8zc"
Aug 02 09:42:42.892626 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:42.892515    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eda18ed-b699-4efd-b283-47c0ac424ecb-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-7b8zc\" (UID: \"9eda18ed-b699-4efd-b283-47c0ac424ecb\") " pod="openshift-ingress/payload-processing-5cc6cd444b-7b8zc"
Aug 02 09:42:42.893178 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:42.893153    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eda18ed-b699-4efd-b283-47c0ac424ecb-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-7b8zc\" (UID: \"9eda18ed-b699-4efd-b283-47c0ac424ecb\") " pod="openshift-ingress/payload-processing-5cc6cd444b-7b8zc"
Aug 02 09:42:42.916388 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:42.916359    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-hfw8f"
Aug 02 09:42:42.924242 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:42.924208    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5shgw\" (UniqueName: \"kubernetes.io/projected/9eda18ed-b699-4efd-b283-47c0ac424ecb-kube-api-access-5shgw\") pod \"payload-processing-5cc6cd444b-7b8zc\" (UID: \"9eda18ed-b699-4efd-b283-47c0ac424ecb\") " pod="openshift-ingress/payload-processing-5cc6cd444b-7b8zc"
Aug 02 09:42:43.026202 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:43.026170    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-856cd5f99f-rv47z" event={"ID":"c674e553-e960-4aa7-8ccf-ef3da49ffa4f","Type":"ContainerStarted","Data":"908b1c58bcefcfcd0ea70a8b5593252e84489fef3900a61df2bf1a91b3092ee2"}
Aug 02 09:42:43.029445 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:43.029419    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-7b8zc"
Aug 02 09:42:43.084701 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:43.084339    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-hfw8f"]
Aug 02 09:42:43.090069 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:42:43.090039    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd41871d5_cf2c_4676_8d98_96060bef7b10.slice/crio-e04b5c8a1035c482f6b20060dd6dca9d51a29a0ebea2ad8148c0991f8d2c6de5 WatchSource:0}: Error finding container e04b5c8a1035c482f6b20060dd6dca9d51a29a0ebea2ad8148c0991f8d2c6de5: Status 404 returned error can't find the container with id e04b5c8a1035c482f6b20060dd6dca9d51a29a0ebea2ad8148c0991f8d2c6de5
Aug 02 09:42:43.179648 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:43.179613    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-7b8zc"]
Aug 02 09:42:43.183896 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:42:43.183873    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eda18ed_b699_4efd_b283_47c0ac424ecb.slice/crio-34895b0e4691051dbdc42b958a38fbefb12f241c25ad66ec797682d41ed84f96 WatchSource:0}: Error finding container 34895b0e4691051dbdc42b958a38fbefb12f241c25ad66ec797682d41ed84f96: Status 404 returned error can't find the container with id 34895b0e4691051dbdc42b958a38fbefb12f241c25ad66ec797682d41ed84f96
Aug 02 09:42:44.035553 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:44.035517    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-7b8zc" event={"ID":"9eda18ed-b699-4efd-b283-47c0ac424ecb","Type":"ContainerStarted","Data":"34895b0e4691051dbdc42b958a38fbefb12f241c25ad66ec797682d41ed84f96"}
Aug 02 09:42:44.037573 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:44.037500    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-hfw8f" event={"ID":"d41871d5-cf2c-4676-8d98-96060bef7b10","Type":"ContainerStarted","Data":"e04b5c8a1035c482f6b20060dd6dca9d51a29a0ebea2ad8148c0991f8d2c6de5"}
Aug 02 09:42:45.043445 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:45.043404    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-856cd5f99f-rv47z" event={"ID":"c674e553-e960-4aa7-8ccf-ef3da49ffa4f","Type":"ContainerStarted","Data":"f76f25452af08accbece289f0374724df518dab0f0247d4424222954ebc5df24"}
Aug 02 09:42:45.043889 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:45.043494    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-856cd5f99f-rv47z"
Aug 02 09:42:46.049060 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:46.048970    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-hfw8f" event={"ID":"d41871d5-cf2c-4676-8d98-96060bef7b10","Type":"ContainerStarted","Data":"0dbf2e8e0bedcc9c8a1630cc3426543605f473f5f01bf185a2b1d9b976948a47"}
Aug 02 09:42:46.049509 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:46.049129    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-hfw8f"
Aug 02 09:42:46.050524 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:46.050500    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-7b8zc" event={"ID":"9eda18ed-b699-4efd-b283-47c0ac424ecb","Type":"ContainerStarted","Data":"83e2bf2ad7cfb23e1c0656eaaafa0b094c35567c0b6737dfad20b98405e9e5b2"}
Aug 02 09:42:46.093138 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:46.093096    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-856cd5f99f-rv47z" podStartSLOduration=3.428174291 podStartE2EDuration="5.093082503s" totalImagesPullingTime="1.664908212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 09:42:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:42:45.064820917 +0000 UTC m=+588.779626365" watchObservedRunningTime="2026-08-02 09:42:46.093082503 +0000 UTC m=+589.807887950"
Aug 02 09:42:46.125724 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:46.125663    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-hfw8f" podStartSLOduration=1.51592167 podStartE2EDuration="4.125644041s" totalImagesPullingTime="2.609722371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 09:42:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:42:46.093208143 +0000 UTC m=+589.808013569" watchObservedRunningTime="2026-08-02 09:42:46.125644041 +0000 UTC m=+589.840449489"
Aug 02 09:42:46.126075 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:46.126046    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-7b8zc" podStartSLOduration=1.6038513559999998 podStartE2EDuration="4.126038875s" totalImagesPullingTime="2.522187519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 09:42:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:42:46.12190698 +0000 UTC m=+589.836712522" watchObservedRunningTime="2026-08-02 09:42:46.126038875 +0000 UTC m=+589.840844328"
Aug 02 09:42:47.053967 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:47.053930    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-7b8zc"
Aug 02 09:42:51.055414 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:51.055388    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-856cd5f99f-rv47z"
Aug 02 09:42:52.147562 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:52.147531    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:42:52.282066 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:52.282019    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:42:57.056529 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:57.056500    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-hfw8f"
Aug 02 09:42:57.254170 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:57.254142    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b4629"]
Aug 02 09:42:57.257310 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:57.257293    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b4629"
Aug 02 09:42:57.265322 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:57.265302    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Aug 02 09:42:57.265418 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:57.265370    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Aug 02 09:42:57.265418 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:57.265371    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Aug 02 09:42:57.266155 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:57.266142    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-jh287\""
Aug 02 09:42:57.290849 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:57.290820    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b4629"]
Aug 02 09:42:57.303184 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:57.303153    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt9vl\" (UniqueName: \"kubernetes.io/projected/e7e35e27-82ae-4072-b8e8-fe60a8a08bfc-kube-api-access-qt9vl\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-b4629\" (UID: \"e7e35e27-82ae-4072-b8e8-fe60a8a08bfc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b4629"
Aug 02 09:42:57.303296 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:57.303189    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7e35e27-82ae-4072-b8e8-fe60a8a08bfc-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-b4629\" (UID: \"e7e35e27-82ae-4072-b8e8-fe60a8a08bfc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b4629"
Aug 02 09:42:57.303296 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:57.303220    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7e35e27-82ae-4072-b8e8-fe60a8a08bfc-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-b4629\" (UID: \"e7e35e27-82ae-4072-b8e8-fe60a8a08bfc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b4629"
Aug 02 09:42:57.303296 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:57.303274    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7e35e27-82ae-4072-b8e8-fe60a8a08bfc-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-b4629\" (UID: \"e7e35e27-82ae-4072-b8e8-fe60a8a08bfc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b4629"
Aug 02 09:42:57.303296 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:57.303293    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7e35e27-82ae-4072-b8e8-fe60a8a08bfc-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-b4629\" (UID: \"e7e35e27-82ae-4072-b8e8-fe60a8a08bfc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b4629"
Aug 02 09:42:57.303457 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:57.303313    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7e35e27-82ae-4072-b8e8-fe60a8a08bfc-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-b4629\" (UID: \"e7e35e27-82ae-4072-b8e8-fe60a8a08bfc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b4629"
Aug 02 09:42:57.331934 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:57.331865    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:42:57.404017 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:57.403987    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7e35e27-82ae-4072-b8e8-fe60a8a08bfc-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-b4629\" (UID: \"e7e35e27-82ae-4072-b8e8-fe60a8a08bfc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b4629"
Aug 02 09:42:57.404017 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:57.404022    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7e35e27-82ae-4072-b8e8-fe60a8a08bfc-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-b4629\" (UID: \"e7e35e27-82ae-4072-b8e8-fe60a8a08bfc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b4629"
Aug 02 09:42:57.404213 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:57.404040    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7e35e27-82ae-4072-b8e8-fe60a8a08bfc-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-b4629\" (UID: \"e7e35e27-82ae-4072-b8e8-fe60a8a08bfc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b4629"
Aug 02 09:42:57.404213 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:57.404066    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7e35e27-82ae-4072-b8e8-fe60a8a08bfc-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-b4629\" (UID: \"e7e35e27-82ae-4072-b8e8-fe60a8a08bfc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b4629"
Aug 02 09:42:57.404213 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:57.404114    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt9vl\" (UniqueName: \"kubernetes.io/projected/e7e35e27-82ae-4072-b8e8-fe60a8a08bfc-kube-api-access-qt9vl\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-b4629\" (UID: \"e7e35e27-82ae-4072-b8e8-fe60a8a08bfc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b4629"
Aug 02 09:42:57.404213 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:57.404150    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7e35e27-82ae-4072-b8e8-fe60a8a08bfc-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-b4629\" (UID: \"e7e35e27-82ae-4072-b8e8-fe60a8a08bfc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b4629"
Aug 02 09:42:57.404487 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:57.404461    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7e35e27-82ae-4072-b8e8-fe60a8a08bfc-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-b4629\" (UID: \"e7e35e27-82ae-4072-b8e8-fe60a8a08bfc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b4629"
Aug 02 09:42:57.404549 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:57.404513    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7e35e27-82ae-4072-b8e8-fe60a8a08bfc-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-b4629\" (UID: \"e7e35e27-82ae-4072-b8e8-fe60a8a08bfc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b4629"
Aug 02 09:42:57.404587 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:57.404578    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7e35e27-82ae-4072-b8e8-fe60a8a08bfc-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-b4629\" (UID: \"e7e35e27-82ae-4072-b8e8-fe60a8a08bfc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b4629"
Aug 02 09:42:57.406276 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:57.406258    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7e35e27-82ae-4072-b8e8-fe60a8a08bfc-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-b4629\" (UID: \"e7e35e27-82ae-4072-b8e8-fe60a8a08bfc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b4629"
Aug 02 09:42:57.406603 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:57.406583    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7e35e27-82ae-4072-b8e8-fe60a8a08bfc-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-b4629\" (UID: \"e7e35e27-82ae-4072-b8e8-fe60a8a08bfc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b4629"
Aug 02 09:42:57.415102 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:57.415076    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt9vl\" (UniqueName: \"kubernetes.io/projected/e7e35e27-82ae-4072-b8e8-fe60a8a08bfc-kube-api-access-qt9vl\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-b4629\" (UID: \"e7e35e27-82ae-4072-b8e8-fe60a8a08bfc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b4629"
Aug 02 09:42:57.567279 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:57.567228    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b4629"
Aug 02 09:42:57.716769 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:57.716737    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b4629"]
Aug 02 09:42:57.720094 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:42:57.720068    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7e35e27_82ae_4072_b8e8_fe60a8a08bfc.slice/crio-2d664bc521d86d71933225aa62fe7e0dc4d8bd7adb0fe12b5767bf6627a08bde WatchSource:0}: Error finding container 2d664bc521d86d71933225aa62fe7e0dc4d8bd7adb0fe12b5767bf6627a08bde: Status 404 returned error can't find the container with id 2d664bc521d86d71933225aa62fe7e0dc4d8bd7adb0fe12b5767bf6627a08bde
Aug 02 09:42:58.059063 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:58.059031    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-7b8zc"
Aug 02 09:42:58.091967 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:58.091935    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b4629" event={"ID":"e7e35e27-82ae-4072-b8e8-fe60a8a08bfc","Type":"ContainerStarted","Data":"2d664bc521d86d71933225aa62fe7e0dc4d8bd7adb0fe12b5767bf6627a08bde"}
Aug 02 09:42:59.097616 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:42:59.097529    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b4629" event={"ID":"e7e35e27-82ae-4072-b8e8-fe60a8a08bfc","Type":"ContainerStarted","Data":"37e21233f53e02b5d99da0a28a0a4f89cb0e6ed3777b14a58576917238427223"}
Aug 02 09:43:03.456869 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:43:03.456837    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:43:07.568305 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:43:07.568190    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b4629"
Aug 02 09:43:07.568305 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:43:07.568274    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b4629"
Aug 02 09:43:07.581282 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:43:07.581254    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b4629"
Aug 02 09:43:07.613148 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:43:07.613095    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b4629" podStartSLOduration=9.639550412 podStartE2EDuration="10.613076379s" totalImagesPullingTime="973.525967ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 09:42:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:42:59.130637603 +0000 UTC m=+602.845443050" watchObservedRunningTime="2026-08-02 09:43:07.613076379 +0000 UTC m=+611.327881827"
Aug 02 09:43:07.656525 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:43:07.656491    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:43:08.145707 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:43:08.145679    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b4629"
Aug 02 09:43:09.343352 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:43:09.343322    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:43:21.098049 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:43:21.097966    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flp55n"]
Aug 02 09:43:21.101786 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:43:21.101767    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flp55n"
Aug 02 09:43:21.106586 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:43:21.106563    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Aug 02 09:43:21.124931 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:43:21.124905    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flp55n"]
Aug 02 09:43:21.171098 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:43:21.171073    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:43:21.176360 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:43:21.176335    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95023ac0-e83b-4b4c-9db9-de898589f429-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flp55n\" (UID: \"95023ac0-e83b-4b4c-9db9-de898589f429\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flp55n"
Aug 02 09:43:21.176442 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:43:21.176369    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95023ac0-e83b-4b4c-9db9-de898589f429-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flp55n\" (UID: \"95023ac0-e83b-4b4c-9db9-de898589f429\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flp55n"
Aug 02 09:43:21.176442 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:43:21.176392    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95023ac0-e83b-4b4c-9db9-de898589f429-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flp55n\" (UID: \"95023ac0-e83b-4b4c-9db9-de898589f429\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flp55n"
Aug 02 09:43:21.176442 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:43:21.176412    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95023ac0-e83b-4b4c-9db9-de898589f429-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flp55n\" (UID: \"95023ac0-e83b-4b4c-9db9-de898589f429\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flp55n"
Aug 02 09:43:21.176442 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:43:21.176431    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95023ac0-e83b-4b4c-9db9-de898589f429-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flp55n\" (UID: \"95023ac0-e83b-4b4c-9db9-de898589f429\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flp55n"
Aug 02 09:43:21.176594 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:43:21.176474    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr88n\" (UniqueName: \"kubernetes.io/projected/95023ac0-e83b-4b4c-9db9-de898589f429-kube-api-access-rr88n\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flp55n\" (UID: \"95023ac0-e83b-4b4c-9db9-de898589f429\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flp55n"
Aug 02 09:43:21.277619 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:43:21.277580    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95023ac0-e83b-4b4c-9db9-de898589f429-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flp55n\" (UID: \"95023ac0-e83b-4b4c-9db9-de898589f429\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flp55n"
Aug 02 09:43:21.277619 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:43:21.277628    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95023ac0-e83b-4b4c-9db9-de898589f429-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flp55n\" (UID: \"95023ac0-e83b-4b4c-9db9-de898589f429\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flp55n"
Aug 02 09:43:21.277835 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:43:21.277649    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95023ac0-e83b-4b4c-9db9-de898589f429-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flp55n\" (UID: \"95023ac0-e83b-4b4c-9db9-de898589f429\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flp55n"
Aug 02 09:43:21.277835 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:43:21.277671    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95023ac0-e83b-4b4c-9db9-de898589f429-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flp55n\" (UID: \"95023ac0-e83b-4b4c-9db9-de898589f429\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flp55n"
Aug 02 09:43:21.277835 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:43:21.277696    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95023ac0-e83b-4b4c-9db9-de898589f429-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flp55n\" (UID: \"95023ac0-e83b-4b4c-9db9-de898589f429\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flp55n"
Aug 02 09:43:21.277835 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:43:21.277714    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr88n\" (UniqueName: \"kubernetes.io/projected/95023ac0-e83b-4b4c-9db9-de898589f429-kube-api-access-rr88n\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flp55n\" (UID: \"95023ac0-e83b-4b4c-9db9-de898589f429\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flp55n"
Aug 02 09:43:21.278038 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:43:21.277993    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95023ac0-e83b-4b4c-9db9-de898589f429-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flp55n\" (UID: \"95023ac0-e83b-4b4c-9db9-de898589f429\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flp55n"
Aug 02 09:43:21.278098 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:43:21.278063    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95023ac0-e83b-4b4c-9db9-de898589f429-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flp55n\" (UID: \"95023ac0-e83b-4b4c-9db9-de898589f429\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flp55n"
Aug 02 09:43:21.278141 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:43:21.278113    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95023ac0-e83b-4b4c-9db9-de898589f429-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flp55n\" (UID: \"95023ac0-e83b-4b4c-9db9-de898589f429\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flp55n"
Aug 02 09:43:21.279893 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:43:21.279873    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95023ac0-e83b-4b4c-9db9-de898589f429-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flp55n\" (UID: \"95023ac0-e83b-4b4c-9db9-de898589f429\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flp55n"
Aug 02 09:43:21.280154 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:43:21.280135    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95023ac0-e83b-4b4c-9db9-de898589f429-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flp55n\" (UID: \"95023ac0-e83b-4b4c-9db9-de898589f429\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flp55n"
Aug 02 09:43:21.286271 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:43:21.286248    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr88n\" (UniqueName: \"kubernetes.io/projected/95023ac0-e83b-4b4c-9db9-de898589f429-kube-api-access-rr88n\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flp55n\" (UID: \"95023ac0-e83b-4b4c-9db9-de898589f429\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flp55n"
Aug 02 09:43:21.410830 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:43:21.410802    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flp55n"
Aug 02 09:43:21.539570 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:43:21.539544    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flp55n"]
Aug 02 09:43:21.540947 ip-10-0-141-232 kubenswrapper[2564]: W0802 09:43:21.540923    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95023ac0_e83b_4b4c_9db9_de898589f429.slice/crio-0fae55b85d3661afe0b34e0479e031f025de294f177b452724d14a5545556644 WatchSource:0}: Error finding container 0fae55b85d3661afe0b34e0479e031f025de294f177b452724d14a5545556644: Status 404 returned error can't find the container with id 0fae55b85d3661afe0b34e0479e031f025de294f177b452724d14a5545556644
Aug 02 09:43:22.184967 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:43:22.184925    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flp55n" event={"ID":"95023ac0-e83b-4b4c-9db9-de898589f429","Type":"ContainerStarted","Data":"265b2591b904d6ec7ce319e67f8c75ee69725c06a98656ba7955e8627dc9f95c"}
Aug 02 09:43:22.185320 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:43:22.185014    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flp55n" event={"ID":"95023ac0-e83b-4b4c-9db9-de898589f429","Type":"ContainerStarted","Data":"0fae55b85d3661afe0b34e0479e031f025de294f177b452724d14a5545556644"}
Aug 02 09:43:22.233503 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:43:22.233461    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flp55n" podStartSLOduration=1.074784111 podStartE2EDuration="1.23344719s" totalImagesPullingTime="158.663079ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 09:43:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:43:22.232390773 +0000 UTC m=+625.947196223" watchObservedRunningTime="2026-08-02 09:43:22.23344719 +0000 UTC m=+625.948252638"
Aug 02 09:43:31.411579 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:43:31.411545    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flp55n"
Aug 02 09:43:31.412032 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:43:31.411603    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flp55n"
Aug 02 09:43:31.423978 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:43:31.423947    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flp55n"
Aug 02 09:43:32.235542 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:43:32.235511    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flp55n"
Aug 02 09:43:49.223488 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:43:49.223447    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:43:58.227570 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:43:58.227530    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:44:09.840784 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:44:09.840739    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:44:10.025133 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:44:10.025091    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:44:48.802137 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:44:48.802099    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:44:48.902755 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:44:48.902725    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:44:50.002248 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:44:50.002209    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:44:50.101314 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:44:50.101282    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:45:13.950442 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:45:13.950411    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:45:19.218037 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:45:19.217998    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:45:19.318414 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:45:19.318381    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:45:50.779027 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:45:50.778994    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:46:30.856847 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:46:30.856744    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:46:54.983423 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:46:54.983385    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:46:56.243141 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:46:56.243105    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:46:59.554828 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:46:59.554797    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:46:59.607141 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:46:59.607097    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:47:57.202855 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:47:57.202822    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:47:57.301676 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:47:57.301642    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:47:57.405938 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:47:57.405905    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:48:03.063219 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:48:03.063172    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:48:03.123728 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:48:03.123699    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:48:03.214862 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:48:03.214826    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:48:03.313958 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:48:03.313852    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:48:03.417966 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:48:03.417936    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:48:09.160708 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:48:09.160671    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:48:09.228857 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:48:09.228815    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:50:44.313606 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:50:44.313570    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:52:48.904134 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:52:48.904104    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:55:12.363374 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:55:12.363337    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:57:16.068207 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:57:16.068122    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:57:29.255657 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:57:29.255619    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:57:50.582764 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:57:50.582717    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:57:50.682251 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:57:50.682182    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:57:51.691584 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:57:51.691546    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:59:11.077484 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:59:11.077453    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:59:11.130445 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:59:11.130398    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:59:25.034883 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:59:25.034850    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 09:59:25.135145 ip-10-0-141-232 kubenswrapper[2564]: I0802 09:59:25.135116    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:01:15.367192 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:15.367160    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:01:41.468603 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:41.468573    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:01:44.499455 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:44.499408    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:01:44.597157 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:44.597116    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:01:44.950353 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:44.950322    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:01:50.744497 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:50.744450    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:01:50.807866 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:50.807831    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:01:50.978625 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:50.978586    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:01:51.278266 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:51.278211    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:01:51.452536 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:51.452505    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:01:54.025533 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:54.025470    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:01:54.124441 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:54.124414    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:01:55.200225 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:55.200186    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ea933196-758cbcd8b7-zgsm9"]
Aug 02 10:01:55.203509 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:55.203491    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea933196-758cbcd8b7-zgsm9"
Aug 02 10:01:55.205900 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:55.205880    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ea933196-serving-cert\""
Aug 02 10:01:55.213415 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:55.213391    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ea933196-758cbcd8b7-zgsm9"]
Aug 02 10:01:55.233706 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:55.233672    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:01:55.263770 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:55.263748    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc7a3906-4392-4d4d-8a4c-cd78210eeae2-maas-api-tls\") pod \"maas-api-e2e-ait-ea933196-758cbcd8b7-zgsm9\" (UID: \"cc7a3906-4392-4d4d-8a4c-cd78210eeae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea933196-758cbcd8b7-zgsm9"
Aug 02 10:01:55.263868 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:55.263810    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh2dr\" (UniqueName: \"kubernetes.io/projected/cc7a3906-4392-4d4d-8a4c-cd78210eeae2-kube-api-access-sh2dr\") pod \"maas-api-e2e-ait-ea933196-758cbcd8b7-zgsm9\" (UID: \"cc7a3906-4392-4d4d-8a4c-cd78210eeae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea933196-758cbcd8b7-zgsm9"
Aug 02 10:01:55.263868 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:55.263842    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc7a3906-4392-4d4d-8a4c-cd78210eeae2-openshift-service-ca\") pod \"maas-api-e2e-ait-ea933196-758cbcd8b7-zgsm9\" (UID: \"cc7a3906-4392-4d4d-8a4c-cd78210eeae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea933196-758cbcd8b7-zgsm9"
Aug 02 10:01:55.365038 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:55.365013    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc7a3906-4392-4d4d-8a4c-cd78210eeae2-maas-api-tls\") pod \"maas-api-e2e-ait-ea933196-758cbcd8b7-zgsm9\" (UID: \"cc7a3906-4392-4d4d-8a4c-cd78210eeae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea933196-758cbcd8b7-zgsm9"
Aug 02 10:01:55.365182 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:55.365053    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh2dr\" (UniqueName: \"kubernetes.io/projected/cc7a3906-4392-4d4d-8a4c-cd78210eeae2-kube-api-access-sh2dr\") pod \"maas-api-e2e-ait-ea933196-758cbcd8b7-zgsm9\" (UID: \"cc7a3906-4392-4d4d-8a4c-cd78210eeae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea933196-758cbcd8b7-zgsm9"
Aug 02 10:01:55.365182 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:55.365077    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc7a3906-4392-4d4d-8a4c-cd78210eeae2-openshift-service-ca\") pod \"maas-api-e2e-ait-ea933196-758cbcd8b7-zgsm9\" (UID: \"cc7a3906-4392-4d4d-8a4c-cd78210eeae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea933196-758cbcd8b7-zgsm9"
Aug 02 10:01:55.365182 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:01:55.365158    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-ea933196-serving-cert: secret "maas-api-e2e-ait-ea933196-serving-cert" not found
Aug 02 10:01:55.365356 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:01:55.365248    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc7a3906-4392-4d4d-8a4c-cd78210eeae2-maas-api-tls podName:cc7a3906-4392-4d4d-8a4c-cd78210eeae2 nodeName:}" failed. No retries permitted until 2026-08-02 10:01:55.865217455 +0000 UTC m=+1739.580022886 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cc7a3906-4392-4d4d-8a4c-cd78210eeae2-maas-api-tls") pod "maas-api-e2e-ait-ea933196-758cbcd8b7-zgsm9" (UID: "cc7a3906-4392-4d4d-8a4c-cd78210eeae2") : secret "maas-api-e2e-ait-ea933196-serving-cert" not found
Aug 02 10:01:55.365733 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:55.365715    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc7a3906-4392-4d4d-8a4c-cd78210eeae2-openshift-service-ca\") pod \"maas-api-e2e-ait-ea933196-758cbcd8b7-zgsm9\" (UID: \"cc7a3906-4392-4d4d-8a4c-cd78210eeae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea933196-758cbcd8b7-zgsm9"
Aug 02 10:01:55.373629 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:55.373605    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh2dr\" (UniqueName: \"kubernetes.io/projected/cc7a3906-4392-4d4d-8a4c-cd78210eeae2-kube-api-access-sh2dr\") pod \"maas-api-e2e-ait-ea933196-758cbcd8b7-zgsm9\" (UID: \"cc7a3906-4392-4d4d-8a4c-cd78210eeae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea933196-758cbcd8b7-zgsm9"
Aug 02 10:01:55.424506 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:55.424475    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:01:55.766093 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:55.766058    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:01:55.868057 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:55.868024    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc7a3906-4392-4d4d-8a4c-cd78210eeae2-maas-api-tls\") pod \"maas-api-e2e-ait-ea933196-758cbcd8b7-zgsm9\" (UID: \"cc7a3906-4392-4d4d-8a4c-cd78210eeae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea933196-758cbcd8b7-zgsm9"
Aug 02 10:01:55.868802 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:55.868749    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:01:55.870393 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:55.870369    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc7a3906-4392-4d4d-8a4c-cd78210eeae2-maas-api-tls\") pod \"maas-api-e2e-ait-ea933196-758cbcd8b7-zgsm9\" (UID: \"cc7a3906-4392-4d4d-8a4c-cd78210eeae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea933196-758cbcd8b7-zgsm9"
Aug 02 10:01:55.877848 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:55.877829    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ea933196-5d584f67dd-9hsc9"]
Aug 02 10:01:55.881148 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:55.881134    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ea933196-5d584f67dd-9hsc9"
Aug 02 10:01:55.884543 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:55.884524    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ea933196\""
Aug 02 10:01:55.884636 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:55.884596    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ea933196-dockercfg-r9fpw\""
Aug 02 10:01:55.895326 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:55.895304    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ea933196-5d584f67dd-9hsc9"]
Aug 02 10:01:55.909304 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:55.909276    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ea933196-645d8d48db-55pvd"]
Aug 02 10:01:55.912372 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:55.912356    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ea933196-645d8d48db-55pvd"
Aug 02 10:01:55.925633 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:55.925611    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ea933196-645d8d48db-55pvd"]
Aug 02 10:01:55.969426 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:55.969401    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7l5w\" (UniqueName: \"kubernetes.io/projected/e15a7aac-6358-4782-8bf5-b1d166c8b2aa-kube-api-access-g7l5w\") pod \"payload-pre-processing-e2e-ait-ea933196-5d584f67dd-9hsc9\" (UID: \"e15a7aac-6358-4782-8bf5-b1d166c8b2aa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ea933196-5d584f67dd-9hsc9"
Aug 02 10:01:55.969526 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:55.969433    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e15a7aac-6358-4782-8bf5-b1d166c8b2aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ea933196-5d584f67dd-9hsc9\" (UID: \"e15a7aac-6358-4782-8bf5-b1d166c8b2aa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ea933196-5d584f67dd-9hsc9"
Aug 02 10:01:56.070124 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:56.070052    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7l5w\" (UniqueName: \"kubernetes.io/projected/e15a7aac-6358-4782-8bf5-b1d166c8b2aa-kube-api-access-g7l5w\") pod \"payload-pre-processing-e2e-ait-ea933196-5d584f67dd-9hsc9\" (UID: \"e15a7aac-6358-4782-8bf5-b1d166c8b2aa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ea933196-5d584f67dd-9hsc9"
Aug 02 10:01:56.070124 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:56.070089    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcm5j\" (UniqueName: \"kubernetes.io/projected/c9d8549c-c6b0-4452-8aef-cba120d9dbd8-kube-api-access-vcm5j\") pod \"payload-processing-e2e-ait-ea933196-645d8d48db-55pvd\" (UID: \"c9d8549c-c6b0-4452-8aef-cba120d9dbd8\") " pod="openshift-ingress/payload-processing-e2e-ait-ea933196-645d8d48db-55pvd"
Aug 02 10:01:56.070325 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:56.070210    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e15a7aac-6358-4782-8bf5-b1d166c8b2aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ea933196-5d584f67dd-9hsc9\" (UID: \"e15a7aac-6358-4782-8bf5-b1d166c8b2aa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ea933196-5d584f67dd-9hsc9"
Aug 02 10:01:56.070325 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:56.070284    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9d8549c-c6b0-4452-8aef-cba120d9dbd8-plugins-config-volume\") pod \"payload-processing-e2e-ait-ea933196-645d8d48db-55pvd\" (UID: \"c9d8549c-c6b0-4452-8aef-cba120d9dbd8\") " pod="openshift-ingress/payload-processing-e2e-ait-ea933196-645d8d48db-55pvd"
Aug 02 10:01:56.070881 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:56.070864    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e15a7aac-6358-4782-8bf5-b1d166c8b2aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ea933196-5d584f67dd-9hsc9\" (UID: \"e15a7aac-6358-4782-8bf5-b1d166c8b2aa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ea933196-5d584f67dd-9hsc9"
Aug 02 10:01:56.081341 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:56.081319    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7l5w\" (UniqueName: \"kubernetes.io/projected/e15a7aac-6358-4782-8bf5-b1d166c8b2aa-kube-api-access-g7l5w\") pod \"payload-pre-processing-e2e-ait-ea933196-5d584f67dd-9hsc9\" (UID: \"e15a7aac-6358-4782-8bf5-b1d166c8b2aa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ea933196-5d584f67dd-9hsc9"
Aug 02 10:01:56.115279 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:56.115255    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea933196-758cbcd8b7-zgsm9"
Aug 02 10:01:56.170867 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:56.170836    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9d8549c-c6b0-4452-8aef-cba120d9dbd8-plugins-config-volume\") pod \"payload-processing-e2e-ait-ea933196-645d8d48db-55pvd\" (UID: \"c9d8549c-c6b0-4452-8aef-cba120d9dbd8\") " pod="openshift-ingress/payload-processing-e2e-ait-ea933196-645d8d48db-55pvd"
Aug 02 10:01:56.171015 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:56.170937    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcm5j\" (UniqueName: \"kubernetes.io/projected/c9d8549c-c6b0-4452-8aef-cba120d9dbd8-kube-api-access-vcm5j\") pod \"payload-processing-e2e-ait-ea933196-645d8d48db-55pvd\" (UID: \"c9d8549c-c6b0-4452-8aef-cba120d9dbd8\") " pod="openshift-ingress/payload-processing-e2e-ait-ea933196-645d8d48db-55pvd"
Aug 02 10:01:56.171698 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:56.171658    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9d8549c-c6b0-4452-8aef-cba120d9dbd8-plugins-config-volume\") pod \"payload-processing-e2e-ait-ea933196-645d8d48db-55pvd\" (UID: \"c9d8549c-c6b0-4452-8aef-cba120d9dbd8\") " pod="openshift-ingress/payload-processing-e2e-ait-ea933196-645d8d48db-55pvd"
Aug 02 10:01:56.178904 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:56.178865    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcm5j\" (UniqueName: \"kubernetes.io/projected/c9d8549c-c6b0-4452-8aef-cba120d9dbd8-kube-api-access-vcm5j\") pod \"payload-processing-e2e-ait-ea933196-645d8d48db-55pvd\" (UID: \"c9d8549c-c6b0-4452-8aef-cba120d9dbd8\") " pod="openshift-ingress/payload-processing-e2e-ait-ea933196-645d8d48db-55pvd"
Aug 02 10:01:56.191139 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:56.190962    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ea933196-5d584f67dd-9hsc9"
Aug 02 10:01:56.222060 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:56.222030    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ea933196-645d8d48db-55pvd"
Aug 02 10:01:56.244737 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:56.244709    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ea933196-758cbcd8b7-zgsm9"]
Aug 02 10:01:56.245647 ip-10-0-141-232 kubenswrapper[2564]: W0802 10:01:56.245616    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc7a3906_4392_4d4d_8a4c_cd78210eeae2.slice/crio-d8d664d39bb90bf2dae68a00149d666347f0ed52dd0d2f9da337b7275d52f4c9 WatchSource:0}: Error finding container d8d664d39bb90bf2dae68a00149d666347f0ed52dd0d2f9da337b7275d52f4c9: Status 404 returned error can't find the container with id d8d664d39bb90bf2dae68a00149d666347f0ed52dd0d2f9da337b7275d52f4c9
Aug 02 10:01:56.248053 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:56.248034    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 02 10:01:56.331587 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:56.331160    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ea933196-5d584f67dd-9hsc9"]
Aug 02 10:01:56.335121 ip-10-0-141-232 kubenswrapper[2564]: W0802 10:01:56.335076    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode15a7aac_6358_4782_8bf5_b1d166c8b2aa.slice/crio-b13ffca9fe8b45e15ab10673388fcc33139078c65e7ccdb3e40ea3b2c04eae72 WatchSource:0}: Error finding container b13ffca9fe8b45e15ab10673388fcc33139078c65e7ccdb3e40ea3b2c04eae72: Status 404 returned error can't find the container with id b13ffca9fe8b45e15ab10673388fcc33139078c65e7ccdb3e40ea3b2c04eae72
Aug 02 10:01:56.356971 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:56.356946    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ea933196-645d8d48db-55pvd"]
Aug 02 10:01:56.366301 ip-10-0-141-232 kubenswrapper[2564]: W0802 10:01:56.366276    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9d8549c_c6b0_4452_8aef_cba120d9dbd8.slice/crio-fa4d92e3524e4fe9a1e6b9d7c447217f857b0edf7c1648eb6b99b92224f7214f WatchSource:0}: Error finding container fa4d92e3524e4fe9a1e6b9d7c447217f857b0edf7c1648eb6b99b92224f7214f: Status 404 returned error can't find the container with id fa4d92e3524e4fe9a1e6b9d7c447217f857b0edf7c1648eb6b99b92224f7214f
Aug 02 10:01:56.491421 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:56.491389    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ea933196-645d8d48db-55pvd" event={"ID":"c9d8549c-c6b0-4452-8aef-cba120d9dbd8","Type":"ContainerStarted","Data":"6c3e7dcb63303cb35897ba4329e3e60095823d84e6fcd0560242105720a2f1a9"}
Aug 02 10:01:56.491601 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:56.491449    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ea933196-645d8d48db-55pvd" event={"ID":"c9d8549c-c6b0-4452-8aef-cba120d9dbd8","Type":"ContainerStarted","Data":"fa4d92e3524e4fe9a1e6b9d7c447217f857b0edf7c1648eb6b99b92224f7214f"}
Aug 02 10:01:56.491601 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:56.491479    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ea933196-645d8d48db-55pvd"
Aug 02 10:01:56.492507 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:56.492482    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea933196-758cbcd8b7-zgsm9" event={"ID":"cc7a3906-4392-4d4d-8a4c-cd78210eeae2","Type":"ContainerStarted","Data":"d8d664d39bb90bf2dae68a00149d666347f0ed52dd0d2f9da337b7275d52f4c9"}
Aug 02 10:01:56.493691 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:56.493674    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ea933196-5d584f67dd-9hsc9" event={"ID":"e15a7aac-6358-4782-8bf5-b1d166c8b2aa","Type":"ContainerStarted","Data":"2f233d574d708022a09deec645781209d557764fd171e17d97c2889fc09ad906"}
Aug 02 10:01:56.493775 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:56.493705    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ea933196-5d584f67dd-9hsc9" event={"ID":"e15a7aac-6358-4782-8bf5-b1d166c8b2aa","Type":"ContainerStarted","Data":"b13ffca9fe8b45e15ab10673388fcc33139078c65e7ccdb3e40ea3b2c04eae72"}
Aug 02 10:01:56.493886 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:56.493807    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ea933196-5d584f67dd-9hsc9"
Aug 02 10:01:56.507375 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:56.507341    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ea933196-645d8d48db-55pvd" podStartSLOduration=1.507328197 podStartE2EDuration="1.507328197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:01:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:01:56.505894586 +0000 UTC m=+1740.220700035" watchObservedRunningTime="2026-08-02 10:01:56.507328197 +0000 UTC m=+1740.222133623"
Aug 02 10:01:56.521516 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:56.521477    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-ea933196-5d584f67dd-9hsc9" podStartSLOduration=1.521465343 podStartE2EDuration="1.521465343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:01:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:01:56.519777027 +0000 UTC m=+1740.234582478" watchObservedRunningTime="2026-08-02 10:01:56.521465343 +0000 UTC m=+1740.236270790"
Aug 02 10:01:57.498902 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:57.498857    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea933196-758cbcd8b7-zgsm9" event={"ID":"cc7a3906-4392-4d4d-8a4c-cd78210eeae2","Type":"ContainerStarted","Data":"1e30b81231332138d273191e51250dc47e092e7531836f4f090e3ccde28864dc"}
Aug 02 10:01:57.499377 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:57.499173    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea933196-758cbcd8b7-zgsm9"
Aug 02 10:01:57.515278 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:01:57.515219    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea933196-758cbcd8b7-zgsm9" podStartSLOduration=2.052504273 podStartE2EDuration="2.515206002s" totalImagesPullingTime="462.701729ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:01:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:01:57.514457954 +0000 UTC m=+1741.229263392" watchObservedRunningTime="2026-08-02 10:01:57.515206002 +0000 UTC m=+1741.230011450"
Aug 02 10:02:04.512013 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:04.511986    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea933196-758cbcd8b7-zgsm9"
Aug 02 10:02:07.501590 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:07.501559    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ea933196-5d584f67dd-9hsc9"
Aug 02 10:02:07.502046 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:07.501630    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ea933196-645d8d48db-55pvd"
Aug 02 10:02:27.359988 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:27.359949    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ea933196-758cbcd8b7-zgsm9"]
Aug 02 10:02:27.360559 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:27.360215    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea933196-758cbcd8b7-zgsm9" podUID="cc7a3906-4392-4d4d-8a4c-cd78210eeae2" containerName="maas-api" containerID="cri-o://1e30b81231332138d273191e51250dc47e092e7531836f4f090e3ccde28864dc" gracePeriod=30
Aug 02 10:02:27.380404 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:27.380377    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:02:27.414787 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:27.414753    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ea933196-645d8d48db-55pvd"]
Aug 02 10:02:27.415005 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:27.414962    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ea933196-645d8d48db-55pvd" podUID="c9d8549c-c6b0-4452-8aef-cba120d9dbd8" containerName="payload-processing" containerID="cri-o://6c3e7dcb63303cb35897ba4329e3e60095823d84e6fcd0560242105720a2f1a9" gracePeriod=30
Aug 02 10:02:27.421159 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:27.421134    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ea933196-5d584f67dd-9hsc9"]
Aug 02 10:02:27.421376 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:27.421353    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-ea933196-5d584f67dd-9hsc9" podUID="e15a7aac-6358-4782-8bf5-b1d166c8b2aa" containerName="payload-pre-processing" containerID="cri-o://2f233d574d708022a09deec645781209d557764fd171e17d97c2889fc09ad906" gracePeriod=30
Aug 02 10:02:27.499661 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:27.499630    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ait-ea933196-5d584f67dd-9hsc9" podUID="e15a7aac-6358-4782-8bf5-b1d166c8b2aa" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.25:9004: connect: connection refused"
Aug 02 10:02:27.499784 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:27.499630    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ait-ea933196-645d8d48db-55pvd" podUID="c9d8549c-c6b0-4452-8aef-cba120d9dbd8" containerName="payload-processing" probeResult="failure" output="dial tcp 10.133.0.26:9004: connect: connection refused"
Aug 02 10:02:27.626374 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:27.626341    2564 generic.go:356] "Generic (PLEG): container finished" podID="e15a7aac-6358-4782-8bf5-b1d166c8b2aa" containerID="2f233d574d708022a09deec645781209d557764fd171e17d97c2889fc09ad906" exitCode=0
Aug 02 10:02:27.626512 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:27.626418    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ea933196-5d584f67dd-9hsc9" event={"ID":"e15a7aac-6358-4782-8bf5-b1d166c8b2aa","Type":"ContainerDied","Data":"2f233d574d708022a09deec645781209d557764fd171e17d97c2889fc09ad906"}
Aug 02 10:02:27.628140 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:27.628117    2564 generic.go:356] "Generic (PLEG): container finished" podID="c9d8549c-c6b0-4452-8aef-cba120d9dbd8" containerID="6c3e7dcb63303cb35897ba4329e3e60095823d84e6fcd0560242105720a2f1a9" exitCode=0
Aug 02 10:02:27.628258 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:27.628181    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ea933196-645d8d48db-55pvd" event={"ID":"c9d8549c-c6b0-4452-8aef-cba120d9dbd8","Type":"ContainerDied","Data":"6c3e7dcb63303cb35897ba4329e3e60095823d84e6fcd0560242105720a2f1a9"}
Aug 02 10:02:27.629968 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:27.629945    2564 generic.go:356] "Generic (PLEG): container finished" podID="cc7a3906-4392-4d4d-8a4c-cd78210eeae2" containerID="1e30b81231332138d273191e51250dc47e092e7531836f4f090e3ccde28864dc" exitCode=0
Aug 02 10:02:27.630050 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:27.629982    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea933196-758cbcd8b7-zgsm9" event={"ID":"cc7a3906-4392-4d4d-8a4c-cd78210eeae2","Type":"ContainerDied","Data":"1e30b81231332138d273191e51250dc47e092e7531836f4f090e3ccde28864dc"}
Aug 02 10:02:27.665973 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:27.665951    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ea933196-645d8d48db-55pvd"
Aug 02 10:02:27.682455 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:27.682434    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea933196-758cbcd8b7-zgsm9"
Aug 02 10:02:27.715497 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:27.715474    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ea933196-5d584f67dd-9hsc9"
Aug 02 10:02:27.722140 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:27.722123    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc7a3906-4392-4d4d-8a4c-cd78210eeae2-openshift-service-ca\") pod \"cc7a3906-4392-4d4d-8a4c-cd78210eeae2\" (UID: \"cc7a3906-4392-4d4d-8a4c-cd78210eeae2\") "
Aug 02 10:02:27.722276 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:27.722155    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vcm5j\" (UniqueName: \"kubernetes.io/projected/c9d8549c-c6b0-4452-8aef-cba120d9dbd8-kube-api-access-vcm5j\") pod \"c9d8549c-c6b0-4452-8aef-cba120d9dbd8\" (UID: \"c9d8549c-c6b0-4452-8aef-cba120d9dbd8\") "
Aug 02 10:02:27.722276 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:27.722190    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9d8549c-c6b0-4452-8aef-cba120d9dbd8-plugins-config-volume\") pod \"c9d8549c-c6b0-4452-8aef-cba120d9dbd8\" (UID: \"c9d8549c-c6b0-4452-8aef-cba120d9dbd8\") "
Aug 02 10:02:27.722276 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:27.722250    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc7a3906-4392-4d4d-8a4c-cd78210eeae2-maas-api-tls\") pod \"cc7a3906-4392-4d4d-8a4c-cd78210eeae2\" (UID: \"cc7a3906-4392-4d4d-8a4c-cd78210eeae2\") "
Aug 02 10:02:27.722454 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:27.722301    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sh2dr\" (UniqueName: \"kubernetes.io/projected/cc7a3906-4392-4d4d-8a4c-cd78210eeae2-kube-api-access-sh2dr\") pod \"cc7a3906-4392-4d4d-8a4c-cd78210eeae2\" (UID: \"cc7a3906-4392-4d4d-8a4c-cd78210eeae2\") "
Aug 02 10:02:27.722546 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:27.722523    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc7a3906-4392-4d4d-8a4c-cd78210eeae2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cc7a3906-4392-4d4d-8a4c-cd78210eeae2" (UID: "cc7a3906-4392-4d4d-8a4c-cd78210eeae2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 02 10:02:27.722590 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:27.722551    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9d8549c-c6b0-4452-8aef-cba120d9dbd8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c9d8549c-c6b0-4452-8aef-cba120d9dbd8" (UID: "c9d8549c-c6b0-4452-8aef-cba120d9dbd8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 02 10:02:27.724331 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:27.724281    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9d8549c-c6b0-4452-8aef-cba120d9dbd8-kube-api-access-vcm5j" (OuterVolumeSpecName: "kube-api-access-vcm5j") pod "c9d8549c-c6b0-4452-8aef-cba120d9dbd8" (UID: "c9d8549c-c6b0-4452-8aef-cba120d9dbd8"). InnerVolumeSpecName "kube-api-access-vcm5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 02 10:02:27.724413 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:27.724330    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc7a3906-4392-4d4d-8a4c-cd78210eeae2-kube-api-access-sh2dr" (OuterVolumeSpecName: "kube-api-access-sh2dr") pod "cc7a3906-4392-4d4d-8a4c-cd78210eeae2" (UID: "cc7a3906-4392-4d4d-8a4c-cd78210eeae2"). InnerVolumeSpecName "kube-api-access-sh2dr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 02 10:02:27.724413 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:27.724331    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc7a3906-4392-4d4d-8a4c-cd78210eeae2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cc7a3906-4392-4d4d-8a4c-cd78210eeae2" (UID: "cc7a3906-4392-4d4d-8a4c-cd78210eeae2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 02 10:02:27.823740 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:27.823707    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g7l5w\" (UniqueName: \"kubernetes.io/projected/e15a7aac-6358-4782-8bf5-b1d166c8b2aa-kube-api-access-g7l5w\") pod \"e15a7aac-6358-4782-8bf5-b1d166c8b2aa\" (UID: \"e15a7aac-6358-4782-8bf5-b1d166c8b2aa\") "
Aug 02 10:02:27.823891 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:27.823787    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e15a7aac-6358-4782-8bf5-b1d166c8b2aa-plugins-config-volume\") pod \"e15a7aac-6358-4782-8bf5-b1d166c8b2aa\" (UID: \"e15a7aac-6358-4782-8bf5-b1d166c8b2aa\") "
Aug 02 10:02:27.823935 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:27.823907    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh2dr\" (UniqueName: \"kubernetes.io/projected/cc7a3906-4392-4d4d-8a4c-cd78210eeae2-kube-api-access-sh2dr\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:02:27.823935 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:27.823919    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc7a3906-4392-4d4d-8a4c-cd78210eeae2-openshift-service-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:02:27.823935 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:27.823928    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcm5j\" (UniqueName: \"kubernetes.io/projected/c9d8549c-c6b0-4452-8aef-cba120d9dbd8-kube-api-access-vcm5j\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:02:27.824046 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:27.823937    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9d8549c-c6b0-4452-8aef-cba120d9dbd8-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:02:27.824046 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:27.823946    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc7a3906-4392-4d4d-8a4c-cd78210eeae2-maas-api-tls\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:02:27.824135 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:27.824113    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e15a7aac-6358-4782-8bf5-b1d166c8b2aa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e15a7aac-6358-4782-8bf5-b1d166c8b2aa" (UID: "e15a7aac-6358-4782-8bf5-b1d166c8b2aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 02 10:02:27.825680 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:27.825660    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e15a7aac-6358-4782-8bf5-b1d166c8b2aa-kube-api-access-g7l5w" (OuterVolumeSpecName: "kube-api-access-g7l5w") pod "e15a7aac-6358-4782-8bf5-b1d166c8b2aa" (UID: "e15a7aac-6358-4782-8bf5-b1d166c8b2aa"). InnerVolumeSpecName "kube-api-access-g7l5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 02 10:02:27.925292 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:27.925259    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e15a7aac-6358-4782-8bf5-b1d166c8b2aa-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:02:27.925292 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:27.925283    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7l5w\" (UniqueName: \"kubernetes.io/projected/e15a7aac-6358-4782-8bf5-b1d166c8b2aa-kube-api-access-g7l5w\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:02:28.380225 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:28.380200    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:02:28.495250 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:28.495215    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:02:28.641939 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:28.641768    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea933196-758cbcd8b7-zgsm9" event={"ID":"cc7a3906-4392-4d4d-8a4c-cd78210eeae2","Type":"ContainerDied","Data":"d8d664d39bb90bf2dae68a00149d666347f0ed52dd0d2f9da337b7275d52f4c9"}
Aug 02 10:02:28.641939 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:28.641785    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea933196-758cbcd8b7-zgsm9"
Aug 02 10:02:28.641939 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:28.641886    2564 scope.go:117] "RemoveContainer" containerID="1e30b81231332138d273191e51250dc47e092e7531836f4f090e3ccde28864dc"
Aug 02 10:02:28.643561 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:28.643432    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ea933196-5d584f67dd-9hsc9"
Aug 02 10:02:28.643561 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:28.643450    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ea933196-5d584f67dd-9hsc9" event={"ID":"e15a7aac-6358-4782-8bf5-b1d166c8b2aa","Type":"ContainerDied","Data":"b13ffca9fe8b45e15ab10673388fcc33139078c65e7ccdb3e40ea3b2c04eae72"}
Aug 02 10:02:28.648917 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:28.645780    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ea933196-645d8d48db-55pvd" event={"ID":"c9d8549c-c6b0-4452-8aef-cba120d9dbd8","Type":"ContainerDied","Data":"fa4d92e3524e4fe9a1e6b9d7c447217f857b0edf7c1648eb6b99b92224f7214f"}
Aug 02 10:02:28.648917 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:28.645872    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ea933196-645d8d48db-55pvd"
Aug 02 10:02:28.653524 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:28.653452    2564 scope.go:117] "RemoveContainer" containerID="2f233d574d708022a09deec645781209d557764fd171e17d97c2889fc09ad906"
Aug 02 10:02:28.663340 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:28.663321    2564 scope.go:117] "RemoveContainer" containerID="6c3e7dcb63303cb35897ba4329e3e60095823d84e6fcd0560242105720a2f1a9"
Aug 02 10:02:28.666444 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:28.666424    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ea933196-758cbcd8b7-zgsm9"]
Aug 02 10:02:28.672623 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:28.672604    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ea933196-758cbcd8b7-zgsm9"]
Aug 02 10:02:28.681536 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:28.681515    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ea933196-645d8d48db-55pvd"]
Aug 02 10:02:28.690440 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:28.690420    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ea933196-645d8d48db-55pvd"]
Aug 02 10:02:28.705427 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:28.705395    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ea933196-5d584f67dd-9hsc9"]
Aug 02 10:02:28.714274 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:28.714252    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ea933196-5d584f67dd-9hsc9"]
Aug 02 10:02:28.815614 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:28.815579    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:02:28.882562 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:28.882533    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9d8549c-c6b0-4452-8aef-cba120d9dbd8" path="/var/lib/kubelet/pods/c9d8549c-c6b0-4452-8aef-cba120d9dbd8/volumes"
Aug 02 10:02:28.882931 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:28.882917    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc7a3906-4392-4d4d-8a4c-cd78210eeae2" path="/var/lib/kubelet/pods/cc7a3906-4392-4d4d-8a4c-cd78210eeae2/volumes"
Aug 02 10:02:28.883217 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:28.883206    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e15a7aac-6358-4782-8bf5-b1d166c8b2aa" path="/var/lib/kubelet/pods/e15a7aac-6358-4782-8bf5-b1d166c8b2aa/volumes"
Aug 02 10:02:29.020713 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:29.020642    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:02:33.775113 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:33.775078    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:02:33.840492 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:33.840453    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4d0cd833-69958478bd-fngc5"]
Aug 02 10:02:33.840910 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:33.840892    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc7a3906-4392-4d4d-8a4c-cd78210eeae2" containerName="maas-api"
Aug 02 10:02:33.840910 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:33.840911    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc7a3906-4392-4d4d-8a4c-cd78210eeae2" containerName="maas-api"
Aug 02 10:02:33.841050 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:33.840944    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9d8549c-c6b0-4452-8aef-cba120d9dbd8" containerName="payload-processing"
Aug 02 10:02:33.841050 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:33.840953    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9d8549c-c6b0-4452-8aef-cba120d9dbd8" containerName="payload-processing"
Aug 02 10:02:33.841050 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:33.840966    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e15a7aac-6358-4782-8bf5-b1d166c8b2aa" containerName="payload-pre-processing"
Aug 02 10:02:33.841050 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:33.840975    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e15a7aac-6358-4782-8bf5-b1d166c8b2aa" containerName="payload-pre-processing"
Aug 02 10:02:33.841050 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:33.841041    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc7a3906-4392-4d4d-8a4c-cd78210eeae2" containerName="maas-api"
Aug 02 10:02:33.841050 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:33.841051    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e15a7aac-6358-4782-8bf5-b1d166c8b2aa" containerName="payload-pre-processing"
Aug 02 10:02:33.841378 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:33.841062    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9d8549c-c6b0-4452-8aef-cba120d9dbd8" containerName="payload-processing"
Aug 02 10:02:33.844170 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:33.844152    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d0cd833-69958478bd-fngc5"
Aug 02 10:02:33.846688 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:33.846667    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-4d0cd833-serving-cert\""
Aug 02 10:02:33.857959 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:33.857941    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4d0cd833-69958478bd-fngc5"]
Aug 02 10:02:33.871163 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:33.871130    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:02:33.875366 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:33.875341    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d139490d-f49f-4036-a36d-4a2bd45e849a-openshift-service-ca\") pod \"maas-api-e2e-migrate-4d0cd833-69958478bd-fngc5\" (UID: \"d139490d-f49f-4036-a36d-4a2bd45e849a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d0cd833-69958478bd-fngc5"
Aug 02 10:02:33.875515 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:33.875421    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph8nr\" (UniqueName: \"kubernetes.io/projected/d139490d-f49f-4036-a36d-4a2bd45e849a-kube-api-access-ph8nr\") pod \"maas-api-e2e-migrate-4d0cd833-69958478bd-fngc5\" (UID: \"d139490d-f49f-4036-a36d-4a2bd45e849a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d0cd833-69958478bd-fngc5"
Aug 02 10:02:33.875515 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:33.875456    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d139490d-f49f-4036-a36d-4a2bd45e849a-maas-api-tls\") pod \"maas-api-e2e-migrate-4d0cd833-69958478bd-fngc5\" (UID: \"d139490d-f49f-4036-a36d-4a2bd45e849a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d0cd833-69958478bd-fngc5"
Aug 02 10:02:33.976339 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:33.976301    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d139490d-f49f-4036-a36d-4a2bd45e849a-maas-api-tls\") pod \"maas-api-e2e-migrate-4d0cd833-69958478bd-fngc5\" (UID: \"d139490d-f49f-4036-a36d-4a2bd45e849a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d0cd833-69958478bd-fngc5"
Aug 02 10:02:33.976519 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:33.976378    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d139490d-f49f-4036-a36d-4a2bd45e849a-openshift-service-ca\") pod \"maas-api-e2e-migrate-4d0cd833-69958478bd-fngc5\" (UID: \"d139490d-f49f-4036-a36d-4a2bd45e849a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d0cd833-69958478bd-fngc5"
Aug 02 10:02:33.976519 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:33.976421    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph8nr\" (UniqueName: \"kubernetes.io/projected/d139490d-f49f-4036-a36d-4a2bd45e849a-kube-api-access-ph8nr\") pod \"maas-api-e2e-migrate-4d0cd833-69958478bd-fngc5\" (UID: \"d139490d-f49f-4036-a36d-4a2bd45e849a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d0cd833-69958478bd-fngc5"
Aug 02 10:02:33.976519 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:02:33.976471    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-4d0cd833-serving-cert: secret "maas-api-e2e-migrate-4d0cd833-serving-cert" not found
Aug 02 10:02:33.976681 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:02:33.976587    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d139490d-f49f-4036-a36d-4a2bd45e849a-maas-api-tls podName:d139490d-f49f-4036-a36d-4a2bd45e849a nodeName:}" failed. No retries permitted until 2026-08-02 10:02:34.47656472 +0000 UTC m=+1778.191370160 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d139490d-f49f-4036-a36d-4a2bd45e849a-maas-api-tls") pod "maas-api-e2e-migrate-4d0cd833-69958478bd-fngc5" (UID: "d139490d-f49f-4036-a36d-4a2bd45e849a") : secret "maas-api-e2e-migrate-4d0cd833-serving-cert" not found
Aug 02 10:02:33.976977 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:33.976957    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d139490d-f49f-4036-a36d-4a2bd45e849a-openshift-service-ca\") pod \"maas-api-e2e-migrate-4d0cd833-69958478bd-fngc5\" (UID: \"d139490d-f49f-4036-a36d-4a2bd45e849a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d0cd833-69958478bd-fngc5"
Aug 02 10:02:33.986880 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:33.986851    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph8nr\" (UniqueName: \"kubernetes.io/projected/d139490d-f49f-4036-a36d-4a2bd45e849a-kube-api-access-ph8nr\") pod \"maas-api-e2e-migrate-4d0cd833-69958478bd-fngc5\" (UID: \"d139490d-f49f-4036-a36d-4a2bd45e849a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d0cd833-69958478bd-fngc5"
Aug 02 10:02:34.249770 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:34.249734    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:02:34.413054 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:34.413021    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:02:34.480856 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:34.480826    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d139490d-f49f-4036-a36d-4a2bd45e849a-maas-api-tls\") pod \"maas-api-e2e-migrate-4d0cd833-69958478bd-fngc5\" (UID: \"d139490d-f49f-4036-a36d-4a2bd45e849a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d0cd833-69958478bd-fngc5"
Aug 02 10:02:34.483217 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:34.483188    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d139490d-f49f-4036-a36d-4a2bd45e849a-maas-api-tls\") pod \"maas-api-e2e-migrate-4d0cd833-69958478bd-fngc5\" (UID: \"d139490d-f49f-4036-a36d-4a2bd45e849a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d0cd833-69958478bd-fngc5"
Aug 02 10:02:34.515455 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:34.515358    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:02:34.551249 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:34.551208    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4d0cd833-7866c798cc-pzvrs"]
Aug 02 10:02:34.554635 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:34.554622    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4d0cd833-7866c798cc-pzvrs"
Aug 02 10:02:34.556542 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:34.556526    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-4d0cd833\""
Aug 02 10:02:34.565069 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:34.565050    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4d0cd833-7866c798cc-pzvrs"]
Aug 02 10:02:34.607732 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:34.607712    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4d0cd833-65c887d9d8-ww6hx"]
Aug 02 10:02:34.610767 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:34.610752    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d0cd833-65c887d9d8-ww6hx"
Aug 02 10:02:34.612779 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:34.612761    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-4d0cd833-dockercfg-95lp6\""
Aug 02 10:02:34.621064 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:34.621046    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4d0cd833-65c887d9d8-ww6hx"]
Aug 02 10:02:34.682193 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:34.682165    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6dh9\" (UniqueName: \"kubernetes.io/projected/82cd4ff6-c69e-42cc-b2f8-78e186a1345b-kube-api-access-p6dh9\") pod \"payload-pre-processing-e2e-migrate-4d0cd833-65c887d9d8-ww6hx\" (UID: \"82cd4ff6-c69e-42cc-b2f8-78e186a1345b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d0cd833-65c887d9d8-ww6hx"
Aug 02 10:02:34.682346 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:34.682207    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk9pv\" (UniqueName: \"kubernetes.io/projected/c2173b3d-0b51-4071-9c75-098dc8baa36a-kube-api-access-sk9pv\") pod \"payload-processing-e2e-migrate-4d0cd833-7866c798cc-pzvrs\" (UID: \"c2173b3d-0b51-4071-9c75-098dc8baa36a\") " pod="openshift-ingress/payload-processing-e2e-migrate-4d0cd833-7866c798cc-pzvrs"
Aug 02 10:02:34.682346 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:34.682276    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82cd4ff6-c69e-42cc-b2f8-78e186a1345b-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4d0cd833-65c887d9d8-ww6hx\" (UID: \"82cd4ff6-c69e-42cc-b2f8-78e186a1345b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d0cd833-65c887d9d8-ww6hx"
Aug 02 10:02:34.682346 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:34.682317    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2173b3d-0b51-4071-9c75-098dc8baa36a-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4d0cd833-7866c798cc-pzvrs\" (UID: \"c2173b3d-0b51-4071-9c75-098dc8baa36a\") " pod="openshift-ingress/payload-processing-e2e-migrate-4d0cd833-7866c798cc-pzvrs"
Aug 02 10:02:34.759312 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:34.759282    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d0cd833-69958478bd-fngc5"
Aug 02 10:02:34.783347 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:34.783319    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk9pv\" (UniqueName: \"kubernetes.io/projected/c2173b3d-0b51-4071-9c75-098dc8baa36a-kube-api-access-sk9pv\") pod \"payload-processing-e2e-migrate-4d0cd833-7866c798cc-pzvrs\" (UID: \"c2173b3d-0b51-4071-9c75-098dc8baa36a\") " pod="openshift-ingress/payload-processing-e2e-migrate-4d0cd833-7866c798cc-pzvrs"
Aug 02 10:02:34.783752 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:34.783367    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82cd4ff6-c69e-42cc-b2f8-78e186a1345b-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4d0cd833-65c887d9d8-ww6hx\" (UID: \"82cd4ff6-c69e-42cc-b2f8-78e186a1345b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d0cd833-65c887d9d8-ww6hx"
Aug 02 10:02:34.783752 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:34.783393    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2173b3d-0b51-4071-9c75-098dc8baa36a-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4d0cd833-7866c798cc-pzvrs\" (UID: \"c2173b3d-0b51-4071-9c75-098dc8baa36a\") " pod="openshift-ingress/payload-processing-e2e-migrate-4d0cd833-7866c798cc-pzvrs"
Aug 02 10:02:34.783752 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:34.783461    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6dh9\" (UniqueName: \"kubernetes.io/projected/82cd4ff6-c69e-42cc-b2f8-78e186a1345b-kube-api-access-p6dh9\") pod \"payload-pre-processing-e2e-migrate-4d0cd833-65c887d9d8-ww6hx\" (UID: \"82cd4ff6-c69e-42cc-b2f8-78e186a1345b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d0cd833-65c887d9d8-ww6hx"
Aug 02 10:02:34.784292 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:34.784267    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2173b3d-0b51-4071-9c75-098dc8baa36a-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4d0cd833-7866c798cc-pzvrs\" (UID: \"c2173b3d-0b51-4071-9c75-098dc8baa36a\") " pod="openshift-ingress/payload-processing-e2e-migrate-4d0cd833-7866c798cc-pzvrs"
Aug 02 10:02:34.784353 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:34.784295    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82cd4ff6-c69e-42cc-b2f8-78e186a1345b-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4d0cd833-65c887d9d8-ww6hx\" (UID: \"82cd4ff6-c69e-42cc-b2f8-78e186a1345b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d0cd833-65c887d9d8-ww6hx"
Aug 02 10:02:34.791545 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:34.791512    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6dh9\" (UniqueName: \"kubernetes.io/projected/82cd4ff6-c69e-42cc-b2f8-78e186a1345b-kube-api-access-p6dh9\") pod \"payload-pre-processing-e2e-migrate-4d0cd833-65c887d9d8-ww6hx\" (UID: \"82cd4ff6-c69e-42cc-b2f8-78e186a1345b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d0cd833-65c887d9d8-ww6hx"
Aug 02 10:02:34.791658 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:34.791612    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk9pv\" (UniqueName: \"kubernetes.io/projected/c2173b3d-0b51-4071-9c75-098dc8baa36a-kube-api-access-sk9pv\") pod \"payload-processing-e2e-migrate-4d0cd833-7866c798cc-pzvrs\" (UID: \"c2173b3d-0b51-4071-9c75-098dc8baa36a\") " pod="openshift-ingress/payload-processing-e2e-migrate-4d0cd833-7866c798cc-pzvrs"
Aug 02 10:02:34.864798 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:34.864769    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4d0cd833-7866c798cc-pzvrs"
Aug 02 10:02:34.889393 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:34.889367    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4d0cd833-69958478bd-fngc5"]
Aug 02 10:02:34.890875 ip-10-0-141-232 kubenswrapper[2564]: W0802 10:02:34.890853    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd139490d_f49f_4036_a36d_4a2bd45e849a.slice/crio-eca7365e4967206c291413bf88feed5b85bd25b34553e958df9a77fc0f696042 WatchSource:0}: Error finding container eca7365e4967206c291413bf88feed5b85bd25b34553e958df9a77fc0f696042: Status 404 returned error can't find the container with id eca7365e4967206c291413bf88feed5b85bd25b34553e958df9a77fc0f696042
Aug 02 10:02:34.919614 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:34.919592    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d0cd833-65c887d9d8-ww6hx"
Aug 02 10:02:35.042523 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:35.042497    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4d0cd833-65c887d9d8-ww6hx"]
Aug 02 10:02:35.044660 ip-10-0-141-232 kubenswrapper[2564]: W0802 10:02:35.044629    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82cd4ff6_c69e_42cc_b2f8_78e186a1345b.slice/crio-479d6321c5f5e7d8b4e3ff69b3d410672f8631bb24e508266e63b625e6285897 WatchSource:0}: Error finding container 479d6321c5f5e7d8b4e3ff69b3d410672f8631bb24e508266e63b625e6285897: Status 404 returned error can't find the container with id 479d6321c5f5e7d8b4e3ff69b3d410672f8631bb24e508266e63b625e6285897
Aug 02 10:02:35.204133 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:35.203962    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4d0cd833-7866c798cc-pzvrs"]
Aug 02 10:02:35.206671 ip-10-0-141-232 kubenswrapper[2564]: W0802 10:02:35.206649    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2173b3d_0b51_4071_9c75_098dc8baa36a.slice/crio-110b7ff56589c5d0419b7ff71c48284d33cc7177c7467e796fec9f49df41f699 WatchSource:0}: Error finding container 110b7ff56589c5d0419b7ff71c48284d33cc7177c7467e796fec9f49df41f699: Status 404 returned error can't find the container with id 110b7ff56589c5d0419b7ff71c48284d33cc7177c7467e796fec9f49df41f699
Aug 02 10:02:35.680028 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:35.679994    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d0cd833-69958478bd-fngc5" event={"ID":"d139490d-f49f-4036-a36d-4a2bd45e849a","Type":"ContainerStarted","Data":"4ae704ef8ea8790920f0599bfa6928acd46ad3bc724ad4c10cb0c051ca4063ea"}
Aug 02 10:02:35.680189 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:35.680057    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d0cd833-69958478bd-fngc5"
Aug 02 10:02:35.680189 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:35.680081    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d0cd833-69958478bd-fngc5" event={"ID":"d139490d-f49f-4036-a36d-4a2bd45e849a","Type":"ContainerStarted","Data":"eca7365e4967206c291413bf88feed5b85bd25b34553e958df9a77fc0f696042"}
Aug 02 10:02:35.681423 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:35.681400    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d0cd833-65c887d9d8-ww6hx" event={"ID":"82cd4ff6-c69e-42cc-b2f8-78e186a1345b","Type":"ContainerStarted","Data":"3a9d6942214f29e085e668334287c9e402c49e38b8597f22998af5e5cb5cb524"}
Aug 02 10:02:35.681498 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:35.681442    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d0cd833-65c887d9d8-ww6hx" event={"ID":"82cd4ff6-c69e-42cc-b2f8-78e186a1345b","Type":"ContainerStarted","Data":"479d6321c5f5e7d8b4e3ff69b3d410672f8631bb24e508266e63b625e6285897"}
Aug 02 10:02:35.681498 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:35.681471    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d0cd833-65c887d9d8-ww6hx"
Aug 02 10:02:35.682791 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:35.682765    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4d0cd833-7866c798cc-pzvrs" event={"ID":"c2173b3d-0b51-4071-9c75-098dc8baa36a","Type":"ContainerStarted","Data":"4d233258081dd01929294b3240611969e6a2b7536a24648789b34bfbb5bbe2d8"}
Aug 02 10:02:35.682881 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:35.682811    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4d0cd833-7866c798cc-pzvrs" event={"ID":"c2173b3d-0b51-4071-9c75-098dc8baa36a","Type":"ContainerStarted","Data":"110b7ff56589c5d0419b7ff71c48284d33cc7177c7467e796fec9f49df41f699"}
Aug 02 10:02:35.682881 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:35.682839    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-4d0cd833-7866c798cc-pzvrs"
Aug 02 10:02:35.697351 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:35.697313    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d0cd833-69958478bd-fngc5" podStartSLOduration=2.384613132 podStartE2EDuration="2.697299973s" totalImagesPullingTime="312.686841ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:02:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:02:35.695866642 +0000 UTC m=+1779.410672091" watchObservedRunningTime="2026-08-02 10:02:35.697299973 +0000 UTC m=+1779.412105421"
Aug 02 10:02:35.710922 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:35.710883    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d0cd833-65c887d9d8-ww6hx" podStartSLOduration=1.710871356 podStartE2EDuration="1.710871356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:02:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:02:35.709439714 +0000 UTC m=+1779.424245166" watchObservedRunningTime="2026-08-02 10:02:35.710871356 +0000 UTC m=+1779.425676781"
Aug 02 10:02:35.723330 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:35.723296    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-4d0cd833-7866c798cc-pzvrs" podStartSLOduration=1.7232849159999999 podStartE2EDuration="1.723284916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:02:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:02:35.722702674 +0000 UTC m=+1779.437508122" watchObservedRunningTime="2026-08-02 10:02:35.723284916 +0000 UTC m=+1779.438090364"
Aug 02 10:02:41.692999 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:41.692968    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d0cd833-69958478bd-fngc5"
Aug 02 10:02:46.689816 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:46.689779    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d0cd833-65c887d9d8-ww6hx"
Aug 02 10:02:46.690209 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:02:46.689948    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-4d0cd833-7866c798cc-pzvrs"
Aug 02 10:03:15.283601 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.283569    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4d0cd833-69958478bd-fngc5"]
Aug 02 10:03:15.284141 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.283808    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d0cd833-69958478bd-fngc5" podUID="d139490d-f49f-4036-a36d-4a2bd45e849a" containerName="maas-api" containerID="cri-o://4ae704ef8ea8790920f0599bfa6928acd46ad3bc724ad4c10cb0c051ca4063ea" gracePeriod=30
Aug 02 10:03:15.313984 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.313962    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:03:15.348594 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.348568    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4d0cd833-7866c798cc-pzvrs"]
Aug 02 10:03:15.348784 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.348760    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-4d0cd833-7866c798cc-pzvrs" podUID="c2173b3d-0b51-4071-9c75-098dc8baa36a" containerName="payload-processing" containerID="cri-o://4d233258081dd01929294b3240611969e6a2b7536a24648789b34bfbb5bbe2d8" gracePeriod=30
Aug 02 10:03:15.355995 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.355967    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4d0cd833-65c887d9d8-ww6hx"]
Aug 02 10:03:15.356212 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.356189    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d0cd833-65c887d9d8-ww6hx" podUID="82cd4ff6-c69e-42cc-b2f8-78e186a1345b" containerName="payload-pre-processing" containerID="cri-o://3a9d6942214f29e085e668334287c9e402c49e38b8597f22998af5e5cb5cb524" gracePeriod=30
Aug 02 10:03:15.417798 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.417769    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:03:15.503026 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.502988    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:03:15.568847 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.568822    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:03:15.646460 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.646410    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4d0cd833-7866c798cc-pzvrs"
Aug 02 10:03:15.648451 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.648433    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d0cd833-65c887d9d8-ww6hx"
Aug 02 10:03:15.671512 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.671487    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:03:15.672929 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.672911    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d0cd833-69958478bd-fngc5"
Aug 02 10:03:15.799894 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.799861    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d139490d-f49f-4036-a36d-4a2bd45e849a-maas-api-tls\") pod \"d139490d-f49f-4036-a36d-4a2bd45e849a\" (UID: \"d139490d-f49f-4036-a36d-4a2bd45e849a\") "
Aug 02 10:03:15.800056 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.799913    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82cd4ff6-c69e-42cc-b2f8-78e186a1345b-plugins-config-volume\") pod \"82cd4ff6-c69e-42cc-b2f8-78e186a1345b\" (UID: \"82cd4ff6-c69e-42cc-b2f8-78e186a1345b\") "
Aug 02 10:03:15.800056 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.799941    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6dh9\" (UniqueName: \"kubernetes.io/projected/82cd4ff6-c69e-42cc-b2f8-78e186a1345b-kube-api-access-p6dh9\") pod \"82cd4ff6-c69e-42cc-b2f8-78e186a1345b\" (UID: \"82cd4ff6-c69e-42cc-b2f8-78e186a1345b\") "
Aug 02 10:03:15.800056 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.799968    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d139490d-f49f-4036-a36d-4a2bd45e849a-openshift-service-ca\") pod \"d139490d-f49f-4036-a36d-4a2bd45e849a\" (UID: \"d139490d-f49f-4036-a36d-4a2bd45e849a\") "
Aug 02 10:03:15.800056 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.800013    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ph8nr\" (UniqueName: \"kubernetes.io/projected/d139490d-f49f-4036-a36d-4a2bd45e849a-kube-api-access-ph8nr\") pod \"d139490d-f49f-4036-a36d-4a2bd45e849a\" (UID: \"d139490d-f49f-4036-a36d-4a2bd45e849a\") "
Aug 02 10:03:15.800056 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.800054    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sk9pv\" (UniqueName: \"kubernetes.io/projected/c2173b3d-0b51-4071-9c75-098dc8baa36a-kube-api-access-sk9pv\") pod \"c2173b3d-0b51-4071-9c75-098dc8baa36a\" (UID: \"c2173b3d-0b51-4071-9c75-098dc8baa36a\") "
Aug 02 10:03:15.800323 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.800071    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2173b3d-0b51-4071-9c75-098dc8baa36a-plugins-config-volume\") pod \"c2173b3d-0b51-4071-9c75-098dc8baa36a\" (UID: \"c2173b3d-0b51-4071-9c75-098dc8baa36a\") "
Aug 02 10:03:15.800415 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.800366    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d139490d-f49f-4036-a36d-4a2bd45e849a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d139490d-f49f-4036-a36d-4a2bd45e849a" (UID: "d139490d-f49f-4036-a36d-4a2bd45e849a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 02 10:03:15.800596 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.800563    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82cd4ff6-c69e-42cc-b2f8-78e186a1345b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "82cd4ff6-c69e-42cc-b2f8-78e186a1345b" (UID: "82cd4ff6-c69e-42cc-b2f8-78e186a1345b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 02 10:03:15.800721 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.800635    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2173b3d-0b51-4071-9c75-098dc8baa36a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c2173b3d-0b51-4071-9c75-098dc8baa36a" (UID: "c2173b3d-0b51-4071-9c75-098dc8baa36a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 02 10:03:15.802115 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.802088    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d139490d-f49f-4036-a36d-4a2bd45e849a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d139490d-f49f-4036-a36d-4a2bd45e849a" (UID: "d139490d-f49f-4036-a36d-4a2bd45e849a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 02 10:03:15.802209 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.802171    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82cd4ff6-c69e-42cc-b2f8-78e186a1345b-kube-api-access-p6dh9" (OuterVolumeSpecName: "kube-api-access-p6dh9") pod "82cd4ff6-c69e-42cc-b2f8-78e186a1345b" (UID: "82cd4ff6-c69e-42cc-b2f8-78e186a1345b"). InnerVolumeSpecName "kube-api-access-p6dh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 02 10:03:15.802209 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.802189    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2173b3d-0b51-4071-9c75-098dc8baa36a-kube-api-access-sk9pv" (OuterVolumeSpecName: "kube-api-access-sk9pv") pod "c2173b3d-0b51-4071-9c75-098dc8baa36a" (UID: "c2173b3d-0b51-4071-9c75-098dc8baa36a"). InnerVolumeSpecName "kube-api-access-sk9pv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 02 10:03:15.802320 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.802209    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d139490d-f49f-4036-a36d-4a2bd45e849a-kube-api-access-ph8nr" (OuterVolumeSpecName: "kube-api-access-ph8nr") pod "d139490d-f49f-4036-a36d-4a2bd45e849a" (UID: "d139490d-f49f-4036-a36d-4a2bd45e849a"). InnerVolumeSpecName "kube-api-access-ph8nr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 02 10:03:15.849169 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.849114    2564 generic.go:356] "Generic (PLEG): container finished" podID="d139490d-f49f-4036-a36d-4a2bd45e849a" containerID="4ae704ef8ea8790920f0599bfa6928acd46ad3bc724ad4c10cb0c051ca4063ea" exitCode=0
Aug 02 10:03:15.849169 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.849157    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d0cd833-69958478bd-fngc5"
Aug 02 10:03:15.849340 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.849206    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d0cd833-69958478bd-fngc5" event={"ID":"d139490d-f49f-4036-a36d-4a2bd45e849a","Type":"ContainerDied","Data":"4ae704ef8ea8790920f0599bfa6928acd46ad3bc724ad4c10cb0c051ca4063ea"}
Aug 02 10:03:15.849340 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.849305    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d0cd833-69958478bd-fngc5" event={"ID":"d139490d-f49f-4036-a36d-4a2bd45e849a","Type":"ContainerDied","Data":"eca7365e4967206c291413bf88feed5b85bd25b34553e958df9a77fc0f696042"}
Aug 02 10:03:15.849340 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.849325    2564 scope.go:117] "RemoveContainer" containerID="4ae704ef8ea8790920f0599bfa6928acd46ad3bc724ad4c10cb0c051ca4063ea"
Aug 02 10:03:15.850422 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.850400    2564 generic.go:356] "Generic (PLEG): container finished" podID="82cd4ff6-c69e-42cc-b2f8-78e186a1345b" containerID="3a9d6942214f29e085e668334287c9e402c49e38b8597f22998af5e5cb5cb524" exitCode=0
Aug 02 10:03:15.850484 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.850456    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d0cd833-65c887d9d8-ww6hx"
Aug 02 10:03:15.850484 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.850472    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d0cd833-65c887d9d8-ww6hx" event={"ID":"82cd4ff6-c69e-42cc-b2f8-78e186a1345b","Type":"ContainerDied","Data":"3a9d6942214f29e085e668334287c9e402c49e38b8597f22998af5e5cb5cb524"}
Aug 02 10:03:15.850588 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.850515    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4d0cd833-65c887d9d8-ww6hx" event={"ID":"82cd4ff6-c69e-42cc-b2f8-78e186a1345b","Type":"ContainerDied","Data":"479d6321c5f5e7d8b4e3ff69b3d410672f8631bb24e508266e63b625e6285897"}
Aug 02 10:03:15.851698 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.851674    2564 generic.go:356] "Generic (PLEG): container finished" podID="c2173b3d-0b51-4071-9c75-098dc8baa36a" containerID="4d233258081dd01929294b3240611969e6a2b7536a24648789b34bfbb5bbe2d8" exitCode=0
Aug 02 10:03:15.851785 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.851711    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4d0cd833-7866c798cc-pzvrs"
Aug 02 10:03:15.851785 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.851716    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4d0cd833-7866c798cc-pzvrs" event={"ID":"c2173b3d-0b51-4071-9c75-098dc8baa36a","Type":"ContainerDied","Data":"4d233258081dd01929294b3240611969e6a2b7536a24648789b34bfbb5bbe2d8"}
Aug 02 10:03:15.851785 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.851755    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4d0cd833-7866c798cc-pzvrs" event={"ID":"c2173b3d-0b51-4071-9c75-098dc8baa36a","Type":"ContainerDied","Data":"110b7ff56589c5d0419b7ff71c48284d33cc7177c7467e796fec9f49df41f699"}
Aug 02 10:03:15.858792 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.858776    2564 scope.go:117] "RemoveContainer" containerID="4ae704ef8ea8790920f0599bfa6928acd46ad3bc724ad4c10cb0c051ca4063ea"
Aug 02 10:03:15.859054 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:03:15.859031    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ae704ef8ea8790920f0599bfa6928acd46ad3bc724ad4c10cb0c051ca4063ea\": container with ID starting with 4ae704ef8ea8790920f0599bfa6928acd46ad3bc724ad4c10cb0c051ca4063ea not found: ID does not exist" containerID="4ae704ef8ea8790920f0599bfa6928acd46ad3bc724ad4c10cb0c051ca4063ea"
Aug 02 10:03:15.859111 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.859068    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ae704ef8ea8790920f0599bfa6928acd46ad3bc724ad4c10cb0c051ca4063ea"} err="failed to get container status \"4ae704ef8ea8790920f0599bfa6928acd46ad3bc724ad4c10cb0c051ca4063ea\": rpc error: code = NotFound desc = could not find container \"4ae704ef8ea8790920f0599bfa6928acd46ad3bc724ad4c10cb0c051ca4063ea\": container with ID starting with 4ae704ef8ea8790920f0599bfa6928acd46ad3bc724ad4c10cb0c051ca4063ea not found: ID does not exist"
Aug 02 10:03:15.859111 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.859083    2564 scope.go:117] "RemoveContainer" containerID="3a9d6942214f29e085e668334287c9e402c49e38b8597f22998af5e5cb5cb524"
Aug 02 10:03:15.867217 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.867201    2564 scope.go:117] "RemoveContainer" containerID="3a9d6942214f29e085e668334287c9e402c49e38b8597f22998af5e5cb5cb524"
Aug 02 10:03:15.867472 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:03:15.867452    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a9d6942214f29e085e668334287c9e402c49e38b8597f22998af5e5cb5cb524\": container with ID starting with 3a9d6942214f29e085e668334287c9e402c49e38b8597f22998af5e5cb5cb524 not found: ID does not exist" containerID="3a9d6942214f29e085e668334287c9e402c49e38b8597f22998af5e5cb5cb524"
Aug 02 10:03:15.867552 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.867484    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a9d6942214f29e085e668334287c9e402c49e38b8597f22998af5e5cb5cb524"} err="failed to get container status \"3a9d6942214f29e085e668334287c9e402c49e38b8597f22998af5e5cb5cb524\": rpc error: code = NotFound desc = could not find container \"3a9d6942214f29e085e668334287c9e402c49e38b8597f22998af5e5cb5cb524\": container with ID starting with 3a9d6942214f29e085e668334287c9e402c49e38b8597f22998af5e5cb5cb524 not found: ID does not exist"
Aug 02 10:03:15.867552 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.867506    2564 scope.go:117] "RemoveContainer" containerID="4d233258081dd01929294b3240611969e6a2b7536a24648789b34bfbb5bbe2d8"
Aug 02 10:03:15.875016 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.874473    2564 scope.go:117] "RemoveContainer" containerID="4d233258081dd01929294b3240611969e6a2b7536a24648789b34bfbb5bbe2d8"
Aug 02 10:03:15.875016 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:03:15.874738    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d233258081dd01929294b3240611969e6a2b7536a24648789b34bfbb5bbe2d8\": container with ID starting with 4d233258081dd01929294b3240611969e6a2b7536a24648789b34bfbb5bbe2d8 not found: ID does not exist" containerID="4d233258081dd01929294b3240611969e6a2b7536a24648789b34bfbb5bbe2d8"
Aug 02 10:03:15.875016 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.874766    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d233258081dd01929294b3240611969e6a2b7536a24648789b34bfbb5bbe2d8"} err="failed to get container status \"4d233258081dd01929294b3240611969e6a2b7536a24648789b34bfbb5bbe2d8\": rpc error: code = NotFound desc = could not find container \"4d233258081dd01929294b3240611969e6a2b7536a24648789b34bfbb5bbe2d8\": container with ID starting with 4d233258081dd01929294b3240611969e6a2b7536a24648789b34bfbb5bbe2d8 not found: ID does not exist"
Aug 02 10:03:15.875560 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.875542    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4d0cd833-69958478bd-fngc5"]
Aug 02 10:03:15.879692 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.879674    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4d0cd833-69958478bd-fngc5"]
Aug 02 10:03:15.889667 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.889649    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4d0cd833-65c887d9d8-ww6hx"]
Aug 02 10:03:15.892785 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.892769    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4d0cd833-65c887d9d8-ww6hx"]
Aug 02 10:03:15.901276 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.901260    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph8nr\" (UniqueName: \"kubernetes.io/projected/d139490d-f49f-4036-a36d-4a2bd45e849a-kube-api-access-ph8nr\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:03:15.901338 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.901280    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk9pv\" (UniqueName: \"kubernetes.io/projected/c2173b3d-0b51-4071-9c75-098dc8baa36a-kube-api-access-sk9pv\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:03:15.901338 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.901292    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2173b3d-0b51-4071-9c75-098dc8baa36a-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:03:15.901338 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.901301    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d139490d-f49f-4036-a36d-4a2bd45e849a-maas-api-tls\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:03:15.901338 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.901310    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82cd4ff6-c69e-42cc-b2f8-78e186a1345b-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:03:15.901338 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.901318    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6dh9\" (UniqueName: \"kubernetes.io/projected/82cd4ff6-c69e-42cc-b2f8-78e186a1345b-kube-api-access-p6dh9\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:03:15.901338 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.901326    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d139490d-f49f-4036-a36d-4a2bd45e849a-openshift-service-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:03:15.904340 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.904321    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4d0cd833-7866c798cc-pzvrs"]
Aug 02 10:03:15.911439 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.911422    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4d0cd833-7866c798cc-pzvrs"]
Aug 02 10:03:15.922666 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:15.922650    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:03:16.149385 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:16.149364    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:03:16.491645 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:16.491558    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:03:16.591715 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:16.591685    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:03:16.881744 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:16.881716    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82cd4ff6-c69e-42cc-b2f8-78e186a1345b" path="/var/lib/kubelet/pods/82cd4ff6-c69e-42cc-b2f8-78e186a1345b/volumes"
Aug 02 10:03:16.882065 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:16.882053    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2173b3d-0b51-4071-9c75-098dc8baa36a" path="/var/lib/kubelet/pods/c2173b3d-0b51-4071-9c75-098dc8baa36a/volumes"
Aug 02 10:03:16.882470 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:16.882452    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d139490d-f49f-4036-a36d-4a2bd45e849a" path="/var/lib/kubelet/pods/d139490d-f49f-4036-a36d-4a2bd45e849a/volumes"
Aug 02 10:03:18.939228 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:18.939195    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:03:19.107030 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:19.106997    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:03:20.005773 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:20.005744    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-973f165e-7bf99f5fdd-bhpk5"]
Aug 02 10:03:20.006247 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:20.006056    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d139490d-f49f-4036-a36d-4a2bd45e849a" containerName="maas-api"
Aug 02 10:03:20.006247 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:20.006068    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d139490d-f49f-4036-a36d-4a2bd45e849a" containerName="maas-api"
Aug 02 10:03:20.006247 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:20.006081    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82cd4ff6-c69e-42cc-b2f8-78e186a1345b" containerName="payload-pre-processing"
Aug 02 10:03:20.006247 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:20.006089    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="82cd4ff6-c69e-42cc-b2f8-78e186a1345b" containerName="payload-pre-processing"
Aug 02 10:03:20.006247 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:20.006098    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2173b3d-0b51-4071-9c75-098dc8baa36a" containerName="payload-processing"
Aug 02 10:03:20.006247 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:20.006104    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2173b3d-0b51-4071-9c75-098dc8baa36a" containerName="payload-processing"
Aug 02 10:03:20.006247 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:20.006156    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2173b3d-0b51-4071-9c75-098dc8baa36a" containerName="payload-processing"
Aug 02 10:03:20.006247 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:20.006167    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="82cd4ff6-c69e-42cc-b2f8-78e186a1345b" containerName="payload-pre-processing"
Aug 02 10:03:20.006247 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:20.006177    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d139490d-f49f-4036-a36d-4a2bd45e849a" containerName="maas-api"
Aug 02 10:03:20.010825 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:20.010796    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-973f165e-7bf99f5fdd-bhpk5"
Aug 02 10:03:20.012950 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:20.012931    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-973f165e-serving-cert\""
Aug 02 10:03:20.023990 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:20.023851    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-973f165e-7bf99f5fdd-bhpk5"]
Aug 02 10:03:20.035617 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:20.035590    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:03:20.132338 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:20.132302    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqrbk\" (UniqueName: \"kubernetes.io/projected/e5212ffd-7500-4a76-9b95-d389603f76ba-kube-api-access-mqrbk\") pod \"maas-api-e2e-ait-973f165e-7bf99f5fdd-bhpk5\" (UID: \"e5212ffd-7500-4a76-9b95-d389603f76ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-973f165e-7bf99f5fdd-bhpk5"
Aug 02 10:03:20.132338 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:20.132343    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5212ffd-7500-4a76-9b95-d389603f76ba-openshift-service-ca\") pod \"maas-api-e2e-ait-973f165e-7bf99f5fdd-bhpk5\" (UID: \"e5212ffd-7500-4a76-9b95-d389603f76ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-973f165e-7bf99f5fdd-bhpk5"
Aug 02 10:03:20.132535 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:20.132374    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5212ffd-7500-4a76-9b95-d389603f76ba-maas-api-tls\") pod \"maas-api-e2e-ait-973f165e-7bf99f5fdd-bhpk5\" (UID: \"e5212ffd-7500-4a76-9b95-d389603f76ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-973f165e-7bf99f5fdd-bhpk5"
Aug 02 10:03:20.138521 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:20.138494    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:03:20.233517 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:20.233487    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5212ffd-7500-4a76-9b95-d389603f76ba-maas-api-tls\") pod \"maas-api-e2e-ait-973f165e-7bf99f5fdd-bhpk5\" (UID: \"e5212ffd-7500-4a76-9b95-d389603f76ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-973f165e-7bf99f5fdd-bhpk5"
Aug 02 10:03:20.233641 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:20.233564    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqrbk\" (UniqueName: \"kubernetes.io/projected/e5212ffd-7500-4a76-9b95-d389603f76ba-kube-api-access-mqrbk\") pod \"maas-api-e2e-ait-973f165e-7bf99f5fdd-bhpk5\" (UID: \"e5212ffd-7500-4a76-9b95-d389603f76ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-973f165e-7bf99f5fdd-bhpk5"
Aug 02 10:03:20.233641 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:20.233591    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5212ffd-7500-4a76-9b95-d389603f76ba-openshift-service-ca\") pod \"maas-api-e2e-ait-973f165e-7bf99f5fdd-bhpk5\" (UID: \"e5212ffd-7500-4a76-9b95-d389603f76ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-973f165e-7bf99f5fdd-bhpk5"
Aug 02 10:03:20.234203 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:20.234186    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5212ffd-7500-4a76-9b95-d389603f76ba-openshift-service-ca\") pod \"maas-api-e2e-ait-973f165e-7bf99f5fdd-bhpk5\" (UID: \"e5212ffd-7500-4a76-9b95-d389603f76ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-973f165e-7bf99f5fdd-bhpk5"
Aug 02 10:03:20.235912 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:20.235894    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5212ffd-7500-4a76-9b95-d389603f76ba-maas-api-tls\") pod \"maas-api-e2e-ait-973f165e-7bf99f5fdd-bhpk5\" (UID: \"e5212ffd-7500-4a76-9b95-d389603f76ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-973f165e-7bf99f5fdd-bhpk5"
Aug 02 10:03:20.241370 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:20.241346    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqrbk\" (UniqueName: \"kubernetes.io/projected/e5212ffd-7500-4a76-9b95-d389603f76ba-kube-api-access-mqrbk\") pod \"maas-api-e2e-ait-973f165e-7bf99f5fdd-bhpk5\" (UID: \"e5212ffd-7500-4a76-9b95-d389603f76ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-973f165e-7bf99f5fdd-bhpk5"
Aug 02 10:03:20.324153 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:20.324071    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-973f165e-7bf99f5fdd-bhpk5"
Aug 02 10:03:20.525612 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:20.525586    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:03:20.622522 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:20.622443    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:03:20.637771 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:20.637739    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-973f165e-747ffd44f6-c76mj"]
Aug 02 10:03:20.642180 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:20.642159    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-973f165e-747ffd44f6-c76mj"
Aug 02 10:03:20.645144 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:20.645126    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-973f165e-dockercfg-dg7v5\""
Aug 02 10:03:20.645527 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:20.645510    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-973f165e\""
Aug 02 10:03:20.650609 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:20.650590    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-973f165e-747ffd44f6-c76mj"]
Aug 02 10:03:20.663132 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:20.663103    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-973f165e-7bf99f5fdd-bhpk5"]
Aug 02 10:03:20.665102 ip-10-0-141-232 kubenswrapper[2564]: W0802 10:03:20.665071    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5212ffd_7500_4a76_9b95_d389603f76ba.slice/crio-ce959a8d37e3f6725f9c8b8380d92ed3926af05ffee537dce0564190cb221756 WatchSource:0}: Error finding container ce959a8d37e3f6725f9c8b8380d92ed3926af05ffee537dce0564190cb221756: Status 404 returned error can't find the container with id ce959a8d37e3f6725f9c8b8380d92ed3926af05ffee537dce0564190cb221756
Aug 02 10:03:20.667989 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:20.667968    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-973f165e-86f5f47956-ln6q7"]
Aug 02 10:03:20.671419 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:20.671400    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-973f165e-86f5f47956-ln6q7"
Aug 02 10:03:20.680051 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:20.680029    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-973f165e-86f5f47956-ln6q7"]
Aug 02 10:03:20.838084 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:20.838052    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09ac0fe4-4b10-4cfe-82c0-a5b749fbbc13-plugins-config-volume\") pod \"payload-processing-e2e-ait-973f165e-86f5f47956-ln6q7\" (UID: \"09ac0fe4-4b10-4cfe-82c0-a5b749fbbc13\") " pod="openshift-ingress/payload-processing-e2e-ait-973f165e-86f5f47956-ln6q7"
Aug 02 10:03:20.838289 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:20.838104    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsnlr\" (UniqueName: \"kubernetes.io/projected/24e2e8f3-d0c6-4f40-b610-7e3843535441-kube-api-access-qsnlr\") pod \"payload-pre-processing-e2e-ait-973f165e-747ffd44f6-c76mj\" (UID: \"24e2e8f3-d0c6-4f40-b610-7e3843535441\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-973f165e-747ffd44f6-c76mj"
Aug 02 10:03:20.838289 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:20.838125    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsjwk\" (UniqueName: \"kubernetes.io/projected/09ac0fe4-4b10-4cfe-82c0-a5b749fbbc13-kube-api-access-wsjwk\") pod \"payload-processing-e2e-ait-973f165e-86f5f47956-ln6q7\" (UID: \"09ac0fe4-4b10-4cfe-82c0-a5b749fbbc13\") " pod="openshift-ingress/payload-processing-e2e-ait-973f165e-86f5f47956-ln6q7"
Aug 02 10:03:20.838289 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:20.838229    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24e2e8f3-d0c6-4f40-b610-7e3843535441-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-973f165e-747ffd44f6-c76mj\" (UID: \"24e2e8f3-d0c6-4f40-b610-7e3843535441\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-973f165e-747ffd44f6-c76mj"
Aug 02 10:03:20.880547 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:20.880508    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-973f165e-7bf99f5fdd-bhpk5" event={"ID":"e5212ffd-7500-4a76-9b95-d389603f76ba","Type":"ContainerStarted","Data":"ce959a8d37e3f6725f9c8b8380d92ed3926af05ffee537dce0564190cb221756"}
Aug 02 10:03:20.938801 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:20.938769    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09ac0fe4-4b10-4cfe-82c0-a5b749fbbc13-plugins-config-volume\") pod \"payload-processing-e2e-ait-973f165e-86f5f47956-ln6q7\" (UID: \"09ac0fe4-4b10-4cfe-82c0-a5b749fbbc13\") " pod="openshift-ingress/payload-processing-e2e-ait-973f165e-86f5f47956-ln6q7"
Aug 02 10:03:20.938943 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:20.938827    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsnlr\" (UniqueName: \"kubernetes.io/projected/24e2e8f3-d0c6-4f40-b610-7e3843535441-kube-api-access-qsnlr\") pod \"payload-pre-processing-e2e-ait-973f165e-747ffd44f6-c76mj\" (UID: \"24e2e8f3-d0c6-4f40-b610-7e3843535441\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-973f165e-747ffd44f6-c76mj"
Aug 02 10:03:20.938943 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:20.938931    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsjwk\" (UniqueName: \"kubernetes.io/projected/09ac0fe4-4b10-4cfe-82c0-a5b749fbbc13-kube-api-access-wsjwk\") pod \"payload-processing-e2e-ait-973f165e-86f5f47956-ln6q7\" (UID: \"09ac0fe4-4b10-4cfe-82c0-a5b749fbbc13\") " pod="openshift-ingress/payload-processing-e2e-ait-973f165e-86f5f47956-ln6q7"
Aug 02 10:03:20.939054 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:20.938995    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24e2e8f3-d0c6-4f40-b610-7e3843535441-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-973f165e-747ffd44f6-c76mj\" (UID: \"24e2e8f3-d0c6-4f40-b610-7e3843535441\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-973f165e-747ffd44f6-c76mj"
Aug 02 10:03:20.939854 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:20.939831    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09ac0fe4-4b10-4cfe-82c0-a5b749fbbc13-plugins-config-volume\") pod \"payload-processing-e2e-ait-973f165e-86f5f47956-ln6q7\" (UID: \"09ac0fe4-4b10-4cfe-82c0-a5b749fbbc13\") " pod="openshift-ingress/payload-processing-e2e-ait-973f165e-86f5f47956-ln6q7"
Aug 02 10:03:20.939901 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:20.939831    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24e2e8f3-d0c6-4f40-b610-7e3843535441-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-973f165e-747ffd44f6-c76mj\" (UID: \"24e2e8f3-d0c6-4f40-b610-7e3843535441\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-973f165e-747ffd44f6-c76mj"
Aug 02 10:03:20.946690 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:20.946662    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsjwk\" (UniqueName: \"kubernetes.io/projected/09ac0fe4-4b10-4cfe-82c0-a5b749fbbc13-kube-api-access-wsjwk\") pod \"payload-processing-e2e-ait-973f165e-86f5f47956-ln6q7\" (UID: \"09ac0fe4-4b10-4cfe-82c0-a5b749fbbc13\") " pod="openshift-ingress/payload-processing-e2e-ait-973f165e-86f5f47956-ln6q7"
Aug 02 10:03:20.946801 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:20.946694    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsnlr\" (UniqueName: \"kubernetes.io/projected/24e2e8f3-d0c6-4f40-b610-7e3843535441-kube-api-access-qsnlr\") pod \"payload-pre-processing-e2e-ait-973f165e-747ffd44f6-c76mj\" (UID: \"24e2e8f3-d0c6-4f40-b610-7e3843535441\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-973f165e-747ffd44f6-c76mj"
Aug 02 10:03:20.954198 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:20.954179    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-973f165e-747ffd44f6-c76mj"
Aug 02 10:03:20.982026 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:20.982004    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-973f165e-86f5f47956-ln6q7"
Aug 02 10:03:21.092997 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:21.092919    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-973f165e-747ffd44f6-c76mj"]
Aug 02 10:03:21.098340 ip-10-0-141-232 kubenswrapper[2564]: W0802 10:03:21.098269    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24e2e8f3_d0c6_4f40_b610_7e3843535441.slice/crio-218cf390362dacdb20ef33405cb2bf23219b294c6a3149fd14b0fd604a30a2b3 WatchSource:0}: Error finding container 218cf390362dacdb20ef33405cb2bf23219b294c6a3149fd14b0fd604a30a2b3: Status 404 returned error can't find the container with id 218cf390362dacdb20ef33405cb2bf23219b294c6a3149fd14b0fd604a30a2b3
Aug 02 10:03:21.324975 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:21.324951    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-973f165e-86f5f47956-ln6q7"]
Aug 02 10:03:21.327805 ip-10-0-141-232 kubenswrapper[2564]: W0802 10:03:21.327768    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09ac0fe4_4b10_4cfe_82c0_a5b749fbbc13.slice/crio-039b54f04c194c7f45efe9c08451597e3b3c0de3a9b51d54c324cede0af8a5e9 WatchSource:0}: Error finding container 039b54f04c194c7f45efe9c08451597e3b3c0de3a9b51d54c324cede0af8a5e9: Status 404 returned error can't find the container with id 039b54f04c194c7f45efe9c08451597e3b3c0de3a9b51d54c324cede0af8a5e9
Aug 02 10:03:21.881885 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:21.881853    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-973f165e-747ffd44f6-c76mj" event={"ID":"24e2e8f3-d0c6-4f40-b610-7e3843535441","Type":"ContainerStarted","Data":"fbfa81bce3f9a14944ddab5f5c16fb169202adcf9877015f9a1c03b3347d228f"}
Aug 02 10:03:21.882043 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:21.881900    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-973f165e-747ffd44f6-c76mj" event={"ID":"24e2e8f3-d0c6-4f40-b610-7e3843535441","Type":"ContainerStarted","Data":"218cf390362dacdb20ef33405cb2bf23219b294c6a3149fd14b0fd604a30a2b3"}
Aug 02 10:03:21.882043 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:21.882033    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-973f165e-747ffd44f6-c76mj"
Aug 02 10:03:21.883380 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:21.883354    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-973f165e-7bf99f5fdd-bhpk5" event={"ID":"e5212ffd-7500-4a76-9b95-d389603f76ba","Type":"ContainerStarted","Data":"df74217c41fe9a5526f1e84ce283ea14b96764e1e4f08147f24649af8ce4b46d"}
Aug 02 10:03:21.883495 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:21.883396    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-973f165e-7bf99f5fdd-bhpk5"
Aug 02 10:03:21.884696 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:21.884676    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-973f165e-86f5f47956-ln6q7" event={"ID":"09ac0fe4-4b10-4cfe-82c0-a5b749fbbc13","Type":"ContainerStarted","Data":"2df0e8c62bbaccc11c420f0f187f5df8789c524fa793cd840b29efcf6f0fbfe6"}
Aug 02 10:03:21.884780 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:21.884715    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-973f165e-86f5f47956-ln6q7" event={"ID":"09ac0fe4-4b10-4cfe-82c0-a5b749fbbc13","Type":"ContainerStarted","Data":"039b54f04c194c7f45efe9c08451597e3b3c0de3a9b51d54c324cede0af8a5e9"}
Aug 02 10:03:21.884780 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:21.884742    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-973f165e-86f5f47956-ln6q7"
Aug 02 10:03:21.911006 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:21.910968    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-973f165e-747ffd44f6-c76mj" podStartSLOduration=1.910955662 podStartE2EDuration="1.910955662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:03:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:03:21.895479977 +0000 UTC m=+1825.610285426" watchObservedRunningTime="2026-08-02 10:03:21.910955662 +0000 UTC m=+1825.625761111"
Aug 02 10:03:21.911098 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:21.911037    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-973f165e-86f5f47956-ln6q7" podStartSLOduration=1.911031823 podStartE2EDuration="1.911031823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:03:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:03:21.909750942 +0000 UTC m=+1825.624556391" watchObservedRunningTime="2026-08-02 10:03:21.911031823 +0000 UTC m=+1825.625837316"
Aug 02 10:03:21.925622 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:21.925578    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-973f165e-7bf99f5fdd-bhpk5" podStartSLOduration=2.425178252 podStartE2EDuration="2.925567219s" totalImagesPullingTime="500.388967ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:03:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:03:21.925412892 +0000 UTC m=+1825.640218353" watchObservedRunningTime="2026-08-02 10:03:21.925567219 +0000 UTC m=+1825.640372667"
Aug 02 10:03:27.893986 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:27.893946    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-973f165e-7bf99f5fdd-bhpk5"
Aug 02 10:03:32.890403 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:32.890373    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-973f165e-747ffd44f6-c76mj"
Aug 02 10:03:32.890785 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:32.890437    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-973f165e-86f5f47956-ln6q7"
Aug 02 10:03:50.678984 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:50.678952    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-973f165e-7bf99f5fdd-bhpk5"]
Aug 02 10:03:50.679502 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:50.679153    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-973f165e-7bf99f5fdd-bhpk5" podUID="e5212ffd-7500-4a76-9b95-d389603f76ba" containerName="maas-api" containerID="cri-o://df74217c41fe9a5526f1e84ce283ea14b96764e1e4f08147f24649af8ce4b46d" gracePeriod=30
Aug 02 10:03:50.712738 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:50.712712    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:03:50.723667 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:50.723638    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-973f165e-86f5f47956-ln6q7"]
Aug 02 10:03:50.723881 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:50.723855    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-973f165e-86f5f47956-ln6q7" podUID="09ac0fe4-4b10-4cfe-82c0-a5b749fbbc13" containerName="payload-processing" containerID="cri-o://2df0e8c62bbaccc11c420f0f187f5df8789c524fa793cd840b29efcf6f0fbfe6" gracePeriod=30
Aug 02 10:03:50.733909 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:50.733876    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-973f165e-747ffd44f6-c76mj"]
Aug 02 10:03:50.734512 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:50.734180    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-973f165e-747ffd44f6-c76mj" podUID="24e2e8f3-d0c6-4f40-b610-7e3843535441" containerName="payload-pre-processing" containerID="cri-o://fbfa81bce3f9a14944ddab5f5c16fb169202adcf9877015f9a1c03b3347d228f" gracePeriod=30
Aug 02 10:03:50.813674 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:50.813638    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:03:50.898443 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:50.898383    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:03:50.995021 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:50.994997    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-973f165e-86f5f47956-ln6q7"
Aug 02 10:03:50.997613 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:50.997593    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-973f165e-7bf99f5fdd-bhpk5"
Aug 02 10:03:51.006872 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:51.006848    2564 generic.go:356] "Generic (PLEG): container finished" podID="24e2e8f3-d0c6-4f40-b610-7e3843535441" containerID="fbfa81bce3f9a14944ddab5f5c16fb169202adcf9877015f9a1c03b3347d228f" exitCode=0
Aug 02 10:03:51.006983 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:51.006923    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-973f165e-747ffd44f6-c76mj" event={"ID":"24e2e8f3-d0c6-4f40-b610-7e3843535441","Type":"ContainerDied","Data":"fbfa81bce3f9a14944ddab5f5c16fb169202adcf9877015f9a1c03b3347d228f"}
Aug 02 10:03:51.008047 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:51.008030    2564 generic.go:356] "Generic (PLEG): container finished" podID="e5212ffd-7500-4a76-9b95-d389603f76ba" containerID="df74217c41fe9a5526f1e84ce283ea14b96764e1e4f08147f24649af8ce4b46d" exitCode=0
Aug 02 10:03:51.008142 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:51.008079    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-973f165e-7bf99f5fdd-bhpk5"
Aug 02 10:03:51.008142 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:51.008101    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-973f165e-7bf99f5fdd-bhpk5" event={"ID":"e5212ffd-7500-4a76-9b95-d389603f76ba","Type":"ContainerDied","Data":"df74217c41fe9a5526f1e84ce283ea14b96764e1e4f08147f24649af8ce4b46d"}
Aug 02 10:03:51.008263 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:51.008145    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-973f165e-7bf99f5fdd-bhpk5" event={"ID":"e5212ffd-7500-4a76-9b95-d389603f76ba","Type":"ContainerDied","Data":"ce959a8d37e3f6725f9c8b8380d92ed3926af05ffee537dce0564190cb221756"}
Aug 02 10:03:51.008263 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:51.008174    2564 scope.go:117] "RemoveContainer" containerID="df74217c41fe9a5526f1e84ce283ea14b96764e1e4f08147f24649af8ce4b46d"
Aug 02 10:03:51.009276 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:51.009258    2564 generic.go:356] "Generic (PLEG): container finished" podID="09ac0fe4-4b10-4cfe-82c0-a5b749fbbc13" containerID="2df0e8c62bbaccc11c420f0f187f5df8789c524fa793cd840b29efcf6f0fbfe6" exitCode=0
Aug 02 10:03:51.009353 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:51.009289    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-973f165e-86f5f47956-ln6q7"
Aug 02 10:03:51.009353 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:51.009324    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-973f165e-86f5f47956-ln6q7" event={"ID":"09ac0fe4-4b10-4cfe-82c0-a5b749fbbc13","Type":"ContainerDied","Data":"2df0e8c62bbaccc11c420f0f187f5df8789c524fa793cd840b29efcf6f0fbfe6"}
Aug 02 10:03:51.009430 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:51.009355    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-973f165e-86f5f47956-ln6q7" event={"ID":"09ac0fe4-4b10-4cfe-82c0-a5b749fbbc13","Type":"ContainerDied","Data":"039b54f04c194c7f45efe9c08451597e3b3c0de3a9b51d54c324cede0af8a5e9"}
Aug 02 10:03:51.018331 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:51.018307    2564 scope.go:117] "RemoveContainer" containerID="df74217c41fe9a5526f1e84ce283ea14b96764e1e4f08147f24649af8ce4b46d"
Aug 02 10:03:51.018706 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:03:51.018594    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df74217c41fe9a5526f1e84ce283ea14b96764e1e4f08147f24649af8ce4b46d\": container with ID starting with df74217c41fe9a5526f1e84ce283ea14b96764e1e4f08147f24649af8ce4b46d not found: ID does not exist" containerID="df74217c41fe9a5526f1e84ce283ea14b96764e1e4f08147f24649af8ce4b46d"
Aug 02 10:03:51.018706 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:51.018624    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df74217c41fe9a5526f1e84ce283ea14b96764e1e4f08147f24649af8ce4b46d"} err="failed to get container status \"df74217c41fe9a5526f1e84ce283ea14b96764e1e4f08147f24649af8ce4b46d\": rpc error: code = NotFound desc = could not find container \"df74217c41fe9a5526f1e84ce283ea14b96764e1e4f08147f24649af8ce4b46d\": container with ID starting with df74217c41fe9a5526f1e84ce283ea14b96764e1e4f08147f24649af8ce4b46d not found: ID does not exist"
Aug 02 10:03:51.018706 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:51.018646    2564 scope.go:117] "RemoveContainer" containerID="2df0e8c62bbaccc11c420f0f187f5df8789c524fa793cd840b29efcf6f0fbfe6"
Aug 02 10:03:51.027202 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:51.027185    2564 scope.go:117] "RemoveContainer" containerID="2df0e8c62bbaccc11c420f0f187f5df8789c524fa793cd840b29efcf6f0fbfe6"
Aug 02 10:03:51.027483 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:03:51.027465    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2df0e8c62bbaccc11c420f0f187f5df8789c524fa793cd840b29efcf6f0fbfe6\": container with ID starting with 2df0e8c62bbaccc11c420f0f187f5df8789c524fa793cd840b29efcf6f0fbfe6 not found: ID does not exist" containerID="2df0e8c62bbaccc11c420f0f187f5df8789c524fa793cd840b29efcf6f0fbfe6"
Aug 02 10:03:51.027577 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:51.027489    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2df0e8c62bbaccc11c420f0f187f5df8789c524fa793cd840b29efcf6f0fbfe6"} err="failed to get container status \"2df0e8c62bbaccc11c420f0f187f5df8789c524fa793cd840b29efcf6f0fbfe6\": rpc error: code = NotFound desc = could not find container \"2df0e8c62bbaccc11c420f0f187f5df8789c524fa793cd840b29efcf6f0fbfe6\": container with ID starting with 2df0e8c62bbaccc11c420f0f187f5df8789c524fa793cd840b29efcf6f0fbfe6 not found: ID does not exist"
Aug 02 10:03:51.028166 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:51.028149    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-973f165e-747ffd44f6-c76mj"
Aug 02 10:03:51.054524 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:51.054501    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5212ffd-7500-4a76-9b95-d389603f76ba-maas-api-tls\") pod \"e5212ffd-7500-4a76-9b95-d389603f76ba\" (UID: \"e5212ffd-7500-4a76-9b95-d389603f76ba\") "
Aug 02 10:03:51.054632 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:51.054565    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24e2e8f3-d0c6-4f40-b610-7e3843535441-plugins-config-volume\") pod \"24e2e8f3-d0c6-4f40-b610-7e3843535441\" (UID: \"24e2e8f3-d0c6-4f40-b610-7e3843535441\") "
Aug 02 10:03:51.054632 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:51.054624    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09ac0fe4-4b10-4cfe-82c0-a5b749fbbc13-plugins-config-volume\") pod \"09ac0fe4-4b10-4cfe-82c0-a5b749fbbc13\" (UID: \"09ac0fe4-4b10-4cfe-82c0-a5b749fbbc13\") "
Aug 02 10:03:51.054753 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:51.054653    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wsjwk\" (UniqueName: \"kubernetes.io/projected/09ac0fe4-4b10-4cfe-82c0-a5b749fbbc13-kube-api-access-wsjwk\") pod \"09ac0fe4-4b10-4cfe-82c0-a5b749fbbc13\" (UID: \"09ac0fe4-4b10-4cfe-82c0-a5b749fbbc13\") "
Aug 02 10:03:51.054753 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:51.054680    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qsnlr\" (UniqueName: \"kubernetes.io/projected/24e2e8f3-d0c6-4f40-b610-7e3843535441-kube-api-access-qsnlr\") pod \"24e2e8f3-d0c6-4f40-b610-7e3843535441\" (UID: \"24e2e8f3-d0c6-4f40-b610-7e3843535441\") "
Aug 02 10:03:51.054753 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:51.054720    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqrbk\" (UniqueName: \"kubernetes.io/projected/e5212ffd-7500-4a76-9b95-d389603f76ba-kube-api-access-mqrbk\") pod \"e5212ffd-7500-4a76-9b95-d389603f76ba\" (UID: \"e5212ffd-7500-4a76-9b95-d389603f76ba\") "
Aug 02 10:03:51.054916 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:51.054754    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5212ffd-7500-4a76-9b95-d389603f76ba-openshift-service-ca\") pod \"e5212ffd-7500-4a76-9b95-d389603f76ba\" (UID: \"e5212ffd-7500-4a76-9b95-d389603f76ba\") "
Aug 02 10:03:51.055275 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:51.055225    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09ac0fe4-4b10-4cfe-82c0-a5b749fbbc13-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "09ac0fe4-4b10-4cfe-82c0-a5b749fbbc13" (UID: "09ac0fe4-4b10-4cfe-82c0-a5b749fbbc13"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 02 10:03:51.055385 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:51.055309    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24e2e8f3-d0c6-4f40-b610-7e3843535441-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "24e2e8f3-d0c6-4f40-b610-7e3843535441" (UID: "24e2e8f3-d0c6-4f40-b610-7e3843535441"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 02 10:03:51.055385 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:51.055315    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5212ffd-7500-4a76-9b95-d389603f76ba-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e5212ffd-7500-4a76-9b95-d389603f76ba" (UID: "e5212ffd-7500-4a76-9b95-d389603f76ba"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 02 10:03:51.056755 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:51.056732    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5212ffd-7500-4a76-9b95-d389603f76ba-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e5212ffd-7500-4a76-9b95-d389603f76ba" (UID: "e5212ffd-7500-4a76-9b95-d389603f76ba"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 02 10:03:51.057200 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:51.057182    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24e2e8f3-d0c6-4f40-b610-7e3843535441-kube-api-access-qsnlr" (OuterVolumeSpecName: "kube-api-access-qsnlr") pod "24e2e8f3-d0c6-4f40-b610-7e3843535441" (UID: "24e2e8f3-d0c6-4f40-b610-7e3843535441"). InnerVolumeSpecName "kube-api-access-qsnlr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 02 10:03:51.057626 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:51.057607    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5212ffd-7500-4a76-9b95-d389603f76ba-kube-api-access-mqrbk" (OuterVolumeSpecName: "kube-api-access-mqrbk") pod "e5212ffd-7500-4a76-9b95-d389603f76ba" (UID: "e5212ffd-7500-4a76-9b95-d389603f76ba"). InnerVolumeSpecName "kube-api-access-mqrbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 02 10:03:51.057687 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:51.057669    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09ac0fe4-4b10-4cfe-82c0-a5b749fbbc13-kube-api-access-wsjwk" (OuterVolumeSpecName: "kube-api-access-wsjwk") pod "09ac0fe4-4b10-4cfe-82c0-a5b749fbbc13" (UID: "09ac0fe4-4b10-4cfe-82c0-a5b749fbbc13"). InnerVolumeSpecName "kube-api-access-wsjwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 02 10:03:51.156369 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:51.156342    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09ac0fe4-4b10-4cfe-82c0-a5b749fbbc13-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:03:51.156369 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:51.156363    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsjwk\" (UniqueName: \"kubernetes.io/projected/09ac0fe4-4b10-4cfe-82c0-a5b749fbbc13-kube-api-access-wsjwk\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:03:51.156369 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:51.156373    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsnlr\" (UniqueName: \"kubernetes.io/projected/24e2e8f3-d0c6-4f40-b610-7e3843535441-kube-api-access-qsnlr\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:03:51.156540 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:51.156381    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqrbk\" (UniqueName: \"kubernetes.io/projected/e5212ffd-7500-4a76-9b95-d389603f76ba-kube-api-access-mqrbk\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:03:51.156540 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:51.156391    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5212ffd-7500-4a76-9b95-d389603f76ba-openshift-service-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:03:51.156540 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:51.156400    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5212ffd-7500-4a76-9b95-d389603f76ba-maas-api-tls\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:03:51.156540 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:51.156409    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24e2e8f3-d0c6-4f40-b610-7e3843535441-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:03:51.354488 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:51.354462    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-973f165e-7bf99f5fdd-bhpk5"]
Aug 02 10:03:51.361116 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:51.361097    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-973f165e-7bf99f5fdd-bhpk5"]
Aug 02 10:03:51.380852 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:51.380833    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-973f165e-86f5f47956-ln6q7"]
Aug 02 10:03:51.388040 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:51.388021    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-973f165e-86f5f47956-ln6q7"]
Aug 02 10:03:52.016032 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:52.015951    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-973f165e-747ffd44f6-c76mj"
Aug 02 10:03:52.016032 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:52.015970    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-973f165e-747ffd44f6-c76mj" event={"ID":"24e2e8f3-d0c6-4f40-b610-7e3843535441","Type":"ContainerDied","Data":"218cf390362dacdb20ef33405cb2bf23219b294c6a3149fd14b0fd604a30a2b3"}
Aug 02 10:03:52.016634 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:52.016031    2564 scope.go:117] "RemoveContainer" containerID="fbfa81bce3f9a14944ddab5f5c16fb169202adcf9877015f9a1c03b3347d228f"
Aug 02 10:03:52.038226 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:52.038205    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-973f165e-747ffd44f6-c76mj"]
Aug 02 10:03:52.040686 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:52.040660    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-973f165e-747ffd44f6-c76mj"]
Aug 02 10:03:52.071836 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:52.071814    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:03:52.287855 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:52.287785    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:03:52.506444 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:52.506418    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:03:52.810930 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:52.810897    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:03:52.880989 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:52.880958    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09ac0fe4-4b10-4cfe-82c0-a5b749fbbc13" path="/var/lib/kubelet/pods/09ac0fe4-4b10-4cfe-82c0-a5b749fbbc13/volumes"
Aug 02 10:03:52.881337 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:52.881323    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24e2e8f3-d0c6-4f40-b610-7e3843535441" path="/var/lib/kubelet/pods/24e2e8f3-d0c6-4f40-b610-7e3843535441/volumes"
Aug 02 10:03:52.881673 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:52.881649    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5212ffd-7500-4a76-9b95-d389603f76ba" path="/var/lib/kubelet/pods/e5212ffd-7500-4a76-9b95-d389603f76ba/volumes"
Aug 02 10:03:53.022749 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:53.022723    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:03:55.084418 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:55.084386    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:03:55.186766 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:55.186737    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:03:56.166149 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:56.166113    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4ad08e62-597d67fd69-zgc6s"]
Aug 02 10:03:56.166645 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:56.166570    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09ac0fe4-4b10-4cfe-82c0-a5b749fbbc13" containerName="payload-processing"
Aug 02 10:03:56.166645 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:56.166588    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="09ac0fe4-4b10-4cfe-82c0-a5b749fbbc13" containerName="payload-processing"
Aug 02 10:03:56.166645 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:56.166601    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24e2e8f3-d0c6-4f40-b610-7e3843535441" containerName="payload-pre-processing"
Aug 02 10:03:56.166645 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:56.166609    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="24e2e8f3-d0c6-4f40-b610-7e3843535441" containerName="payload-pre-processing"
Aug 02 10:03:56.166645 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:56.166633    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5212ffd-7500-4a76-9b95-d389603f76ba" containerName="maas-api"
Aug 02 10:03:56.166645 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:56.166642    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5212ffd-7500-4a76-9b95-d389603f76ba" containerName="maas-api"
Aug 02 10:03:56.166903 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:56.166693    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="24e2e8f3-d0c6-4f40-b610-7e3843535441" containerName="payload-pre-processing"
Aug 02 10:03:56.166903 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:56.166703    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5212ffd-7500-4a76-9b95-d389603f76ba" containerName="maas-api"
Aug 02 10:03:56.166903 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:56.166709    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="09ac0fe4-4b10-4cfe-82c0-a5b749fbbc13" containerName="payload-processing"
Aug 02 10:03:56.171046 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:56.171029    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4ad08e62-597d67fd69-zgc6s"
Aug 02 10:03:56.173982 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:56.173959    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-4ad08e62-serving-cert\""
Aug 02 10:03:56.182805 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:56.182783    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4ad08e62-597d67fd69-zgc6s"]
Aug 02 10:03:56.205664 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:56.205644    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:03:56.300738 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:56.300714    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7accaf2f-b1a9-464d-ba91-493519b79c73-openshift-service-ca\") pod \"maas-api-e2e-derive-4ad08e62-597d67fd69-zgc6s\" (UID: \"7accaf2f-b1a9-464d-ba91-493519b79c73\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4ad08e62-597d67fd69-zgc6s"
Aug 02 10:03:56.300854 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:56.300753    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7accaf2f-b1a9-464d-ba91-493519b79c73-maas-api-tls\") pod \"maas-api-e2e-derive-4ad08e62-597d67fd69-zgc6s\" (UID: \"7accaf2f-b1a9-464d-ba91-493519b79c73\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4ad08e62-597d67fd69-zgc6s"
Aug 02 10:03:56.300854 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:56.300774    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx7pf\" (UniqueName: \"kubernetes.io/projected/7accaf2f-b1a9-464d-ba91-493519b79c73-kube-api-access-rx7pf\") pod \"maas-api-e2e-derive-4ad08e62-597d67fd69-zgc6s\" (UID: \"7accaf2f-b1a9-464d-ba91-493519b79c73\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4ad08e62-597d67fd69-zgc6s"
Aug 02 10:03:56.402067 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:56.402036    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7accaf2f-b1a9-464d-ba91-493519b79c73-openshift-service-ca\") pod \"maas-api-e2e-derive-4ad08e62-597d67fd69-zgc6s\" (UID: \"7accaf2f-b1a9-464d-ba91-493519b79c73\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4ad08e62-597d67fd69-zgc6s"
Aug 02 10:03:56.402171 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:56.402090    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7accaf2f-b1a9-464d-ba91-493519b79c73-maas-api-tls\") pod \"maas-api-e2e-derive-4ad08e62-597d67fd69-zgc6s\" (UID: \"7accaf2f-b1a9-464d-ba91-493519b79c73\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4ad08e62-597d67fd69-zgc6s"
Aug 02 10:03:56.402171 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:56.402130    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx7pf\" (UniqueName: \"kubernetes.io/projected/7accaf2f-b1a9-464d-ba91-493519b79c73-kube-api-access-rx7pf\") pod \"maas-api-e2e-derive-4ad08e62-597d67fd69-zgc6s\" (UID: \"7accaf2f-b1a9-464d-ba91-493519b79c73\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4ad08e62-597d67fd69-zgc6s"
Aug 02 10:03:56.402266 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:03:56.402186    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-4ad08e62-serving-cert: secret "maas-api-e2e-derive-4ad08e62-serving-cert" not found
Aug 02 10:03:56.402310 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:03:56.402283    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7accaf2f-b1a9-464d-ba91-493519b79c73-maas-api-tls podName:7accaf2f-b1a9-464d-ba91-493519b79c73 nodeName:}" failed. No retries permitted until 2026-08-02 10:03:56.902260752 +0000 UTC m=+1860.617066183 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7accaf2f-b1a9-464d-ba91-493519b79c73-maas-api-tls") pod "maas-api-e2e-derive-4ad08e62-597d67fd69-zgc6s" (UID: "7accaf2f-b1a9-464d-ba91-493519b79c73") : secret "maas-api-e2e-derive-4ad08e62-serving-cert" not found
Aug 02 10:03:56.402680 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:56.402663    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7accaf2f-b1a9-464d-ba91-493519b79c73-openshift-service-ca\") pod \"maas-api-e2e-derive-4ad08e62-597d67fd69-zgc6s\" (UID: \"7accaf2f-b1a9-464d-ba91-493519b79c73\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4ad08e62-597d67fd69-zgc6s"
Aug 02 10:03:56.412866 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:56.412845    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx7pf\" (UniqueName: \"kubernetes.io/projected/7accaf2f-b1a9-464d-ba91-493519b79c73-kube-api-access-rx7pf\") pod \"maas-api-e2e-derive-4ad08e62-597d67fd69-zgc6s\" (UID: \"7accaf2f-b1a9-464d-ba91-493519b79c73\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4ad08e62-597d67fd69-zgc6s"
Aug 02 10:03:56.702360 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:56.702335    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:03:56.799104 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:56.799069    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:03:56.799534 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:56.799513    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4ad08e62-548b85476d-9vvcc"]
Aug 02 10:03:56.802903 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:56.802885    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4ad08e62-548b85476d-9vvcc"
Aug 02 10:03:56.805177 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:56.805155    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-4ad08e62\""
Aug 02 10:03:56.813348 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:56.813325    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4ad08e62-548b85476d-9vvcc"]
Aug 02 10:03:56.828599 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:56.828578    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4ad08e62-74c84c95c8-649dl"]
Aug 02 10:03:56.831627 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:56.831611    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4ad08e62-74c84c95c8-649dl"
Aug 02 10:03:56.833831 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:56.833815    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-4ad08e62-dockercfg-m4l9b\""
Aug 02 10:03:56.842111 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:56.842091    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4ad08e62-74c84c95c8-649dl"]
Aug 02 10:03:56.907035 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:56.907009    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpjh2\" (UniqueName: \"kubernetes.io/projected/c1a1c7b2-df89-4f22-a7b7-1d0edba6b351-kube-api-access-rpjh2\") pod \"payload-pre-processing-e2e-derive-4ad08e62-548b85476d-9vvcc\" (UID: \"c1a1c7b2-df89-4f22-a7b7-1d0edba6b351\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4ad08e62-548b85476d-9vvcc"
Aug 02 10:03:56.907156 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:56.907073    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7accaf2f-b1a9-464d-ba91-493519b79c73-maas-api-tls\") pod \"maas-api-e2e-derive-4ad08e62-597d67fd69-zgc6s\" (UID: \"7accaf2f-b1a9-464d-ba91-493519b79c73\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4ad08e62-597d67fd69-zgc6s"
Aug 02 10:03:56.907156 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:56.907136    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv9dr\" (UniqueName: \"kubernetes.io/projected/1ef0920c-d048-4206-9f41-82e3e6452723-kube-api-access-mv9dr\") pod \"payload-processing-e2e-derive-4ad08e62-74c84c95c8-649dl\" (UID: \"1ef0920c-d048-4206-9f41-82e3e6452723\") " pod="openshift-ingress/payload-processing-e2e-derive-4ad08e62-74c84c95c8-649dl"
Aug 02 10:03:56.907290 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:56.907168    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ef0920c-d048-4206-9f41-82e3e6452723-plugins-config-volume\") pod \"payload-processing-e2e-derive-4ad08e62-74c84c95c8-649dl\" (UID: \"1ef0920c-d048-4206-9f41-82e3e6452723\") " pod="openshift-ingress/payload-processing-e2e-derive-4ad08e62-74c84c95c8-649dl"
Aug 02 10:03:56.907290 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:56.907244    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1a1c7b2-df89-4f22-a7b7-1d0edba6b351-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-4ad08e62-548b85476d-9vvcc\" (UID: \"c1a1c7b2-df89-4f22-a7b7-1d0edba6b351\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4ad08e62-548b85476d-9vvcc"
Aug 02 10:03:56.909693 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:56.909668    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7accaf2f-b1a9-464d-ba91-493519b79c73-maas-api-tls\") pod \"maas-api-e2e-derive-4ad08e62-597d67fd69-zgc6s\" (UID: \"7accaf2f-b1a9-464d-ba91-493519b79c73\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4ad08e62-597d67fd69-zgc6s"
Aug 02 10:03:57.007814 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:57.007749    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv9dr\" (UniqueName: \"kubernetes.io/projected/1ef0920c-d048-4206-9f41-82e3e6452723-kube-api-access-mv9dr\") pod \"payload-processing-e2e-derive-4ad08e62-74c84c95c8-649dl\" (UID: \"1ef0920c-d048-4206-9f41-82e3e6452723\") " pod="openshift-ingress/payload-processing-e2e-derive-4ad08e62-74c84c95c8-649dl"
Aug 02 10:03:57.007814 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:57.007781    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ef0920c-d048-4206-9f41-82e3e6452723-plugins-config-volume\") pod \"payload-processing-e2e-derive-4ad08e62-74c84c95c8-649dl\" (UID: \"1ef0920c-d048-4206-9f41-82e3e6452723\") " pod="openshift-ingress/payload-processing-e2e-derive-4ad08e62-74c84c95c8-649dl"
Aug 02 10:03:57.007814 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:57.007813    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1a1c7b2-df89-4f22-a7b7-1d0edba6b351-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-4ad08e62-548b85476d-9vvcc\" (UID: \"c1a1c7b2-df89-4f22-a7b7-1d0edba6b351\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4ad08e62-548b85476d-9vvcc"
Aug 02 10:03:57.008067 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:57.007835    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpjh2\" (UniqueName: \"kubernetes.io/projected/c1a1c7b2-df89-4f22-a7b7-1d0edba6b351-kube-api-access-rpjh2\") pod \"payload-pre-processing-e2e-derive-4ad08e62-548b85476d-9vvcc\" (UID: \"c1a1c7b2-df89-4f22-a7b7-1d0edba6b351\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4ad08e62-548b85476d-9vvcc"
Aug 02 10:03:57.008483 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:57.008461    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ef0920c-d048-4206-9f41-82e3e6452723-plugins-config-volume\") pod \"payload-processing-e2e-derive-4ad08e62-74c84c95c8-649dl\" (UID: \"1ef0920c-d048-4206-9f41-82e3e6452723\") " pod="openshift-ingress/payload-processing-e2e-derive-4ad08e62-74c84c95c8-649dl"
Aug 02 10:03:57.008611 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:57.008591    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1a1c7b2-df89-4f22-a7b7-1d0edba6b351-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-4ad08e62-548b85476d-9vvcc\" (UID: \"c1a1c7b2-df89-4f22-a7b7-1d0edba6b351\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4ad08e62-548b85476d-9vvcc"
Aug 02 10:03:57.015718 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:57.015692    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv9dr\" (UniqueName: \"kubernetes.io/projected/1ef0920c-d048-4206-9f41-82e3e6452723-kube-api-access-mv9dr\") pod \"payload-processing-e2e-derive-4ad08e62-74c84c95c8-649dl\" (UID: \"1ef0920c-d048-4206-9f41-82e3e6452723\") " pod="openshift-ingress/payload-processing-e2e-derive-4ad08e62-74c84c95c8-649dl"
Aug 02 10:03:57.015818 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:57.015797    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpjh2\" (UniqueName: \"kubernetes.io/projected/c1a1c7b2-df89-4f22-a7b7-1d0edba6b351-kube-api-access-rpjh2\") pod \"payload-pre-processing-e2e-derive-4ad08e62-548b85476d-9vvcc\" (UID: \"c1a1c7b2-df89-4f22-a7b7-1d0edba6b351\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4ad08e62-548b85476d-9vvcc"
Aug 02 10:03:57.083079 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:57.083056    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4ad08e62-597d67fd69-zgc6s"
Aug 02 10:03:57.114121 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:57.114100    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4ad08e62-548b85476d-9vvcc"
Aug 02 10:03:57.141247 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:57.141144    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4ad08e62-74c84c95c8-649dl"
Aug 02 10:03:57.228530 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:57.228358    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4ad08e62-597d67fd69-zgc6s"]
Aug 02 10:03:57.231181 ip-10-0-141-232 kubenswrapper[2564]: W0802 10:03:57.231153    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7accaf2f_b1a9_464d_ba91_493519b79c73.slice/crio-c5730ea1350b7bc68e30fe423fdf8665c4364449ffcda9fed3f53541fb3109c6 WatchSource:0}: Error finding container c5730ea1350b7bc68e30fe423fdf8665c4364449ffcda9fed3f53541fb3109c6: Status 404 returned error can't find the container with id c5730ea1350b7bc68e30fe423fdf8665c4364449ffcda9fed3f53541fb3109c6
Aug 02 10:03:57.264687 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:57.264655    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4ad08e62-548b85476d-9vvcc"]
Aug 02 10:03:57.266622 ip-10-0-141-232 kubenswrapper[2564]: W0802 10:03:57.266596    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1a1c7b2_df89_4f22_a7b7_1d0edba6b351.slice/crio-4681ddaf703c21e6a609627934aa656a125231ba06f332fd43ced5016181fd43 WatchSource:0}: Error finding container 4681ddaf703c21e6a609627934aa656a125231ba06f332fd43ced5016181fd43: Status 404 returned error can't find the container with id 4681ddaf703c21e6a609627934aa656a125231ba06f332fd43ced5016181fd43
Aug 02 10:03:57.288673 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:57.288653    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4ad08e62-74c84c95c8-649dl"]
Aug 02 10:03:57.298274 ip-10-0-141-232 kubenswrapper[2564]: W0802 10:03:57.298248    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ef0920c_d048_4206_9f41_82e3e6452723.slice/crio-cc7b63babfb61e0a2ed319f81e18fa5813c992abd1fc7e075f12eae998f73f86 WatchSource:0}: Error finding container cc7b63babfb61e0a2ed319f81e18fa5813c992abd1fc7e075f12eae998f73f86: Status 404 returned error can't find the container with id cc7b63babfb61e0a2ed319f81e18fa5813c992abd1fc7e075f12eae998f73f86
Aug 02 10:03:58.048986 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:58.048900    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4ad08e62-74c84c95c8-649dl" event={"ID":"1ef0920c-d048-4206-9f41-82e3e6452723","Type":"ContainerStarted","Data":"b2e6ffc3ec244cc8cf72ea6ee72a613d8d8b2f6c649c130dfd83de0ec20c1951"}
Aug 02 10:03:58.048986 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:58.048962    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4ad08e62-74c84c95c8-649dl" event={"ID":"1ef0920c-d048-4206-9f41-82e3e6452723","Type":"ContainerStarted","Data":"cc7b63babfb61e0a2ed319f81e18fa5813c992abd1fc7e075f12eae998f73f86"}
Aug 02 10:03:58.049199 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:58.049076    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-4ad08e62-74c84c95c8-649dl"
Aug 02 10:03:58.050528 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:58.050502    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4ad08e62-548b85476d-9vvcc" event={"ID":"c1a1c7b2-df89-4f22-a7b7-1d0edba6b351","Type":"ContainerStarted","Data":"fa19143aab7f1e0e600f0dc8fb45a3294be7051ddf2c37bbaf27ba7a3050802b"}
Aug 02 10:03:58.050641 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:58.050538    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4ad08e62-548b85476d-9vvcc" event={"ID":"c1a1c7b2-df89-4f22-a7b7-1d0edba6b351","Type":"ContainerStarted","Data":"4681ddaf703c21e6a609627934aa656a125231ba06f332fd43ced5016181fd43"}
Aug 02 10:03:58.050641 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:58.050565    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-4ad08e62-548b85476d-9vvcc"
Aug 02 10:03:58.051930 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:58.051908    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4ad08e62-597d67fd69-zgc6s" event={"ID":"7accaf2f-b1a9-464d-ba91-493519b79c73","Type":"ContainerStarted","Data":"5f650f4e3fae31c896c37d2169e68ec72239f021af955a7811ad1d8ad409420e"}
Aug 02 10:03:58.052014 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:58.051950    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4ad08e62-597d67fd69-zgc6s" event={"ID":"7accaf2f-b1a9-464d-ba91-493519b79c73","Type":"ContainerStarted","Data":"c5730ea1350b7bc68e30fe423fdf8665c4364449ffcda9fed3f53541fb3109c6"}
Aug 02 10:03:58.052056 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:58.052034    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4ad08e62-597d67fd69-zgc6s"
Aug 02 10:03:58.064064 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:58.064022    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-4ad08e62-74c84c95c8-649dl" podStartSLOduration=2.064010126 podStartE2EDuration="2.064010126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:03:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:03:58.062967651 +0000 UTC m=+1861.777773109" watchObservedRunningTime="2026-08-02 10:03:58.064010126 +0000 UTC m=+1861.778815574"
Aug 02 10:03:58.077903 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:58.077869    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-4ad08e62-548b85476d-9vvcc" podStartSLOduration=2.077858658 podStartE2EDuration="2.077858658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:03:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:03:58.076353384 +0000 UTC m=+1861.791158832" watchObservedRunningTime="2026-08-02 10:03:58.077858658 +0000 UTC m=+1861.792664105"
Aug 02 10:03:58.093218 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:03:58.093179    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4ad08e62-597d67fd69-zgc6s" podStartSLOduration=1.6605760429999998 podStartE2EDuration="2.093166513s" totalImagesPullingTime="432.59047ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:03:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:03:58.09180647 +0000 UTC m=+1861.806611918" watchObservedRunningTime="2026-08-02 10:03:58.093166513 +0000 UTC m=+1861.807971960"
Aug 02 10:04:04.061965 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:04.061936    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4ad08e62-597d67fd69-zgc6s"
Aug 02 10:04:09.058391 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:09.058361    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-4ad08e62-548b85476d-9vvcc"
Aug 02 10:04:09.058857 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:09.058440    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-4ad08e62-74c84c95c8-649dl"
Aug 02 10:04:26.631431 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:26.631354    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4ad08e62-597d67fd69-zgc6s"]
Aug 02 10:04:26.633815 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:26.631600    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4ad08e62-597d67fd69-zgc6s" podUID="7accaf2f-b1a9-464d-ba91-493519b79c73" containerName="maas-api" containerID="cri-o://5f650f4e3fae31c896c37d2169e68ec72239f021af955a7811ad1d8ad409420e" gracePeriod=30
Aug 02 10:04:26.658122 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:26.658098    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:04:26.679491 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:26.679463    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4ad08e62-74c84c95c8-649dl"]
Aug 02 10:04:26.679740 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:26.679693    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-4ad08e62-74c84c95c8-649dl" podUID="1ef0920c-d048-4206-9f41-82e3e6452723" containerName="payload-processing" containerID="cri-o://b2e6ffc3ec244cc8cf72ea6ee72a613d8d8b2f6c649c130dfd83de0ec20c1951" gracePeriod=30
Aug 02 10:04:26.689825 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:26.689625    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4ad08e62-548b85476d-9vvcc"]
Aug 02 10:04:26.689939 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:26.689841    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-4ad08e62-548b85476d-9vvcc" podUID="c1a1c7b2-df89-4f22-a7b7-1d0edba6b351" containerName="payload-pre-processing" containerID="cri-o://fa19143aab7f1e0e600f0dc8fb45a3294be7051ddf2c37bbaf27ba7a3050802b" gracePeriod=30
Aug 02 10:04:26.758871 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:26.758842    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:04:26.827157 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:26.827071    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:04:26.891593 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:26.891516    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:04:26.933457 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:26.933431    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4ad08e62-597d67fd69-zgc6s"
Aug 02 10:04:26.986047 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:26.986024    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4ad08e62-74c84c95c8-649dl"
Aug 02 10:04:26.987437 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:26.987398    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:04:26.991241 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:26.991212    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4ad08e62-548b85476d-9vvcc"
Aug 02 10:04:27.032942 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:27.032916    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7accaf2f-b1a9-464d-ba91-493519b79c73-openshift-service-ca\") pod \"7accaf2f-b1a9-464d-ba91-493519b79c73\" (UID: \"7accaf2f-b1a9-464d-ba91-493519b79c73\") "
Aug 02 10:04:27.033080 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:27.032973    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7accaf2f-b1a9-464d-ba91-493519b79c73-maas-api-tls\") pod \"7accaf2f-b1a9-464d-ba91-493519b79c73\" (UID: \"7accaf2f-b1a9-464d-ba91-493519b79c73\") "
Aug 02 10:04:27.033080 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:27.033046    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rx7pf\" (UniqueName: \"kubernetes.io/projected/7accaf2f-b1a9-464d-ba91-493519b79c73-kube-api-access-rx7pf\") pod \"7accaf2f-b1a9-464d-ba91-493519b79c73\" (UID: \"7accaf2f-b1a9-464d-ba91-493519b79c73\") "
Aug 02 10:04:27.033342 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:27.033321    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7accaf2f-b1a9-464d-ba91-493519b79c73-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7accaf2f-b1a9-464d-ba91-493519b79c73" (UID: "7accaf2f-b1a9-464d-ba91-493519b79c73"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 02 10:04:27.035111 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:27.035087    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7accaf2f-b1a9-464d-ba91-493519b79c73-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7accaf2f-b1a9-464d-ba91-493519b79c73" (UID: "7accaf2f-b1a9-464d-ba91-493519b79c73"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 02 10:04:27.035227 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:27.035092    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7accaf2f-b1a9-464d-ba91-493519b79c73-kube-api-access-rx7pf" (OuterVolumeSpecName: "kube-api-access-rx7pf") pod "7accaf2f-b1a9-464d-ba91-493519b79c73" (UID: "7accaf2f-b1a9-464d-ba91-493519b79c73"). InnerVolumeSpecName "kube-api-access-rx7pf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 02 10:04:27.133629 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:27.133597    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rpjh2\" (UniqueName: \"kubernetes.io/projected/c1a1c7b2-df89-4f22-a7b7-1d0edba6b351-kube-api-access-rpjh2\") pod \"c1a1c7b2-df89-4f22-a7b7-1d0edba6b351\" (UID: \"c1a1c7b2-df89-4f22-a7b7-1d0edba6b351\") "
Aug 02 10:04:27.133807 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:27.133646    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1a1c7b2-df89-4f22-a7b7-1d0edba6b351-plugins-config-volume\") pod \"c1a1c7b2-df89-4f22-a7b7-1d0edba6b351\" (UID: \"c1a1c7b2-df89-4f22-a7b7-1d0edba6b351\") "
Aug 02 10:04:27.133807 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:27.133695    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mv9dr\" (UniqueName: \"kubernetes.io/projected/1ef0920c-d048-4206-9f41-82e3e6452723-kube-api-access-mv9dr\") pod \"1ef0920c-d048-4206-9f41-82e3e6452723\" (UID: \"1ef0920c-d048-4206-9f41-82e3e6452723\") "
Aug 02 10:04:27.133807 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:27.133713    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ef0920c-d048-4206-9f41-82e3e6452723-plugins-config-volume\") pod \"1ef0920c-d048-4206-9f41-82e3e6452723\" (UID: \"1ef0920c-d048-4206-9f41-82e3e6452723\") "
Aug 02 10:04:27.133983 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:27.133851    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx7pf\" (UniqueName: \"kubernetes.io/projected/7accaf2f-b1a9-464d-ba91-493519b79c73-kube-api-access-rx7pf\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:04:27.133983 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:27.133867    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7accaf2f-b1a9-464d-ba91-493519b79c73-openshift-service-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:04:27.133983 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:27.133882    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7accaf2f-b1a9-464d-ba91-493519b79c73-maas-api-tls\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:04:27.134155 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:27.134116    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1a1c7b2-df89-4f22-a7b7-1d0edba6b351-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c1a1c7b2-df89-4f22-a7b7-1d0edba6b351" (UID: "c1a1c7b2-df89-4f22-a7b7-1d0edba6b351"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 02 10:04:27.134461 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:27.134152    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ef0920c-d048-4206-9f41-82e3e6452723-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1ef0920c-d048-4206-9f41-82e3e6452723" (UID: "1ef0920c-d048-4206-9f41-82e3e6452723"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 02 10:04:27.135747 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:27.135723    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1a1c7b2-df89-4f22-a7b7-1d0edba6b351-kube-api-access-rpjh2" (OuterVolumeSpecName: "kube-api-access-rpjh2") pod "c1a1c7b2-df89-4f22-a7b7-1d0edba6b351" (UID: "c1a1c7b2-df89-4f22-a7b7-1d0edba6b351"). InnerVolumeSpecName "kube-api-access-rpjh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 02 10:04:27.135807 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:27.135773    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ef0920c-d048-4206-9f41-82e3e6452723-kube-api-access-mv9dr" (OuterVolumeSpecName: "kube-api-access-mv9dr") pod "1ef0920c-d048-4206-9f41-82e3e6452723" (UID: "1ef0920c-d048-4206-9f41-82e3e6452723"). InnerVolumeSpecName "kube-api-access-mv9dr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 02 10:04:27.169959 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:27.169899    2564 generic.go:356] "Generic (PLEG): container finished" podID="1ef0920c-d048-4206-9f41-82e3e6452723" containerID="b2e6ffc3ec244cc8cf72ea6ee72a613d8d8b2f6c649c130dfd83de0ec20c1951" exitCode=0
Aug 02 10:04:27.169959 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:27.169927    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4ad08e62-74c84c95c8-649dl" event={"ID":"1ef0920c-d048-4206-9f41-82e3e6452723","Type":"ContainerDied","Data":"b2e6ffc3ec244cc8cf72ea6ee72a613d8d8b2f6c649c130dfd83de0ec20c1951"}
Aug 02 10:04:27.169959 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:27.169954    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4ad08e62-74c84c95c8-649dl"
Aug 02 10:04:27.170152 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:27.169985    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4ad08e62-74c84c95c8-649dl" event={"ID":"1ef0920c-d048-4206-9f41-82e3e6452723","Type":"ContainerDied","Data":"cc7b63babfb61e0a2ed319f81e18fa5813c992abd1fc7e075f12eae998f73f86"}
Aug 02 10:04:27.170152 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:27.170018    2564 scope.go:117] "RemoveContainer" containerID="b2e6ffc3ec244cc8cf72ea6ee72a613d8d8b2f6c649c130dfd83de0ec20c1951"
Aug 02 10:04:27.171290 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:27.171261    2564 generic.go:356] "Generic (PLEG): container finished" podID="c1a1c7b2-df89-4f22-a7b7-1d0edba6b351" containerID="fa19143aab7f1e0e600f0dc8fb45a3294be7051ddf2c37bbaf27ba7a3050802b" exitCode=0
Aug 02 10:04:27.171398 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:27.171368    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4ad08e62-548b85476d-9vvcc"
Aug 02 10:04:27.171460 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:27.171399    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4ad08e62-548b85476d-9vvcc" event={"ID":"c1a1c7b2-df89-4f22-a7b7-1d0edba6b351","Type":"ContainerDied","Data":"fa19143aab7f1e0e600f0dc8fb45a3294be7051ddf2c37bbaf27ba7a3050802b"}
Aug 02 10:04:27.171460 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:27.171440    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4ad08e62-548b85476d-9vvcc" event={"ID":"c1a1c7b2-df89-4f22-a7b7-1d0edba6b351","Type":"ContainerDied","Data":"4681ddaf703c21e6a609627934aa656a125231ba06f332fd43ced5016181fd43"}
Aug 02 10:04:27.172679 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:27.172607    2564 generic.go:356] "Generic (PLEG): container finished" podID="7accaf2f-b1a9-464d-ba91-493519b79c73" containerID="5f650f4e3fae31c896c37d2169e68ec72239f021af955a7811ad1d8ad409420e" exitCode=0
Aug 02 10:04:27.172679 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:27.172628    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4ad08e62-597d67fd69-zgc6s" event={"ID":"7accaf2f-b1a9-464d-ba91-493519b79c73","Type":"ContainerDied","Data":"5f650f4e3fae31c896c37d2169e68ec72239f021af955a7811ad1d8ad409420e"}
Aug 02 10:04:27.172679 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:27.172651    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4ad08e62-597d67fd69-zgc6s"
Aug 02 10:04:27.172679 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:27.172675    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4ad08e62-597d67fd69-zgc6s" event={"ID":"7accaf2f-b1a9-464d-ba91-493519b79c73","Type":"ContainerDied","Data":"c5730ea1350b7bc68e30fe423fdf8665c4364449ffcda9fed3f53541fb3109c6"}
Aug 02 10:04:27.179300 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:27.179279    2564 scope.go:117] "RemoveContainer" containerID="b2e6ffc3ec244cc8cf72ea6ee72a613d8d8b2f6c649c130dfd83de0ec20c1951"
Aug 02 10:04:27.179571 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:04:27.179551    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2e6ffc3ec244cc8cf72ea6ee72a613d8d8b2f6c649c130dfd83de0ec20c1951\": container with ID starting with b2e6ffc3ec244cc8cf72ea6ee72a613d8d8b2f6c649c130dfd83de0ec20c1951 not found: ID does not exist" containerID="b2e6ffc3ec244cc8cf72ea6ee72a613d8d8b2f6c649c130dfd83de0ec20c1951"
Aug 02 10:04:27.179643 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:27.179579    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2e6ffc3ec244cc8cf72ea6ee72a613d8d8b2f6c649c130dfd83de0ec20c1951"} err="failed to get container status \"b2e6ffc3ec244cc8cf72ea6ee72a613d8d8b2f6c649c130dfd83de0ec20c1951\": rpc error: code = NotFound desc = could not find container \"b2e6ffc3ec244cc8cf72ea6ee72a613d8d8b2f6c649c130dfd83de0ec20c1951\": container with ID starting with b2e6ffc3ec244cc8cf72ea6ee72a613d8d8b2f6c649c130dfd83de0ec20c1951 not found: ID does not exist"
Aug 02 10:04:27.179643 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:27.179609    2564 scope.go:117] "RemoveContainer" containerID="fa19143aab7f1e0e600f0dc8fb45a3294be7051ddf2c37bbaf27ba7a3050802b"
Aug 02 10:04:27.188369 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:27.188353    2564 scope.go:117] "RemoveContainer" containerID="fa19143aab7f1e0e600f0dc8fb45a3294be7051ddf2c37bbaf27ba7a3050802b"
Aug 02 10:04:27.188603 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:04:27.188587    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa19143aab7f1e0e600f0dc8fb45a3294be7051ddf2c37bbaf27ba7a3050802b\": container with ID starting with fa19143aab7f1e0e600f0dc8fb45a3294be7051ddf2c37bbaf27ba7a3050802b not found: ID does not exist" containerID="fa19143aab7f1e0e600f0dc8fb45a3294be7051ddf2c37bbaf27ba7a3050802b"
Aug 02 10:04:27.188679 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:27.188611    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa19143aab7f1e0e600f0dc8fb45a3294be7051ddf2c37bbaf27ba7a3050802b"} err="failed to get container status \"fa19143aab7f1e0e600f0dc8fb45a3294be7051ddf2c37bbaf27ba7a3050802b\": rpc error: code = NotFound desc = could not find container \"fa19143aab7f1e0e600f0dc8fb45a3294be7051ddf2c37bbaf27ba7a3050802b\": container with ID starting with fa19143aab7f1e0e600f0dc8fb45a3294be7051ddf2c37bbaf27ba7a3050802b not found: ID does not exist"
Aug 02 10:04:27.188679 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:27.188632    2564 scope.go:117] "RemoveContainer" containerID="5f650f4e3fae31c896c37d2169e68ec72239f021af955a7811ad1d8ad409420e"
Aug 02 10:04:27.191473 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:27.191453    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:04:27.195629 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:27.195619    2564 scope.go:117] "RemoveContainer" containerID="5f650f4e3fae31c896c37d2169e68ec72239f021af955a7811ad1d8ad409420e"
Aug 02 10:04:27.195824 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:27.195806    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4ad08e62-548b85476d-9vvcc"]
Aug 02 10:04:27.195878 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:04:27.195836    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f650f4e3fae31c896c37d2169e68ec72239f021af955a7811ad1d8ad409420e\": container with ID starting with 5f650f4e3fae31c896c37d2169e68ec72239f021af955a7811ad1d8ad409420e not found: ID does not exist" containerID="5f650f4e3fae31c896c37d2169e68ec72239f021af955a7811ad1d8ad409420e"
Aug 02 10:04:27.195878 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:27.195860    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f650f4e3fae31c896c37d2169e68ec72239f021af955a7811ad1d8ad409420e"} err="failed to get container status \"5f650f4e3fae31c896c37d2169e68ec72239f021af955a7811ad1d8ad409420e\": rpc error: code = NotFound desc = could not find container \"5f650f4e3fae31c896c37d2169e68ec72239f021af955a7811ad1d8ad409420e\": container with ID starting with 5f650f4e3fae31c896c37d2169e68ec72239f021af955a7811ad1d8ad409420e not found: ID does not exist"
Aug 02 10:04:27.201600 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:27.201582    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4ad08e62-548b85476d-9vvcc"]
Aug 02 10:04:27.211438 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:27.211418    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4ad08e62-597d67fd69-zgc6s"]
Aug 02 10:04:27.213222 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:27.213205    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4ad08e62-597d67fd69-zgc6s"]
Aug 02 10:04:27.223333 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:27.223312    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4ad08e62-74c84c95c8-649dl"]
Aug 02 10:04:27.225362 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:27.225343    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4ad08e62-74c84c95c8-649dl"]
Aug 02 10:04:27.234457 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:27.234441    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1a1c7b2-df89-4f22-a7b7-1d0edba6b351-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:04:27.234510 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:27.234460    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv9dr\" (UniqueName: \"kubernetes.io/projected/1ef0920c-d048-4206-9f41-82e3e6452723-kube-api-access-mv9dr\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:04:27.234510 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:27.234470    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ef0920c-d048-4206-9f41-82e3e6452723-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:04:27.234510 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:27.234478    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpjh2\" (UniqueName: \"kubernetes.io/projected/c1a1c7b2-df89-4f22-a7b7-1d0edba6b351-kube-api-access-rpjh2\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:04:27.685241 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:27.685202    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:04:27.786216 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:27.786186    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:04:27.883569 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:27.883543    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:04:28.880054 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:28.880022    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ef0920c-d048-4206-9f41-82e3e6452723" path="/var/lib/kubelet/pods/1ef0920c-d048-4206-9f41-82e3e6452723/volumes"
Aug 02 10:04:28.880418 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:28.880406    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7accaf2f-b1a9-464d-ba91-493519b79c73" path="/var/lib/kubelet/pods/7accaf2f-b1a9-464d-ba91-493519b79c73/volumes"
Aug 02 10:04:28.880707 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:28.880694    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1a1c7b2-df89-4f22-a7b7-1d0edba6b351" path="/var/lib/kubelet/pods/c1a1c7b2-df89-4f22-a7b7-1d0edba6b351/volumes"
Aug 02 10:04:39.533622 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:39.533588    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:04:46.417024 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:46.416968    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:04:46.521511 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:46.521479    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:04:46.617298 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:46.617270    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:04:46.719544 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:46.719475    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:04:47.129569 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:04:47.129542    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:05:13.078965 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:05:13.078928    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:05:18.947191 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:05:18.947159    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:05:19.069399 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:05:19.069372    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:05:19.154555 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:05:19.154525    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:05:35.048941 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:05:35.048903    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:05:40.639024 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:05:40.638991    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:05:40.700265 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:05:40.700222    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:05:40.778019 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:05:40.777985    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:05:40.876623 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:05:40.876588    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:05:46.216896 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:05:46.216854    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:05:51.379631 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:05:51.379553    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:06:02.471014 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:02.470979    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:06:07.148371 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:07.148336    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:06:08.495894 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:08.495857    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:06:08.563098 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:08.563068    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:06:08.651757 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:08.651730    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:06:08.750050 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:08.749966    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:06:08.849932 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:08.849886    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:06:09.466952 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:09.466909    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:06:09.600040 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:09.600007    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:06:09.743263 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:09.743157    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:06:11.236425 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:11.236388    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:06:17.497389 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:17.497347    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:06:17.598055 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:17.598021    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:06:18.190737 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:18.190701    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:06:24.466051 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:24.466018    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:06:24.567119 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:24.567085    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:06:24.708452 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:24.708421    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:06:25.027940 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:25.027905    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:06:25.693654 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:25.693600    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:06:28.456531 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:28.456491    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:06:28.559421 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:28.559386    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:06:29.540197 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:29.540142    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-756eaa56-fb6cf6795-nsnjb"]
Aug 02 10:06:29.540656 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:29.540582    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1a1c7b2-df89-4f22-a7b7-1d0edba6b351" containerName="payload-pre-processing"
Aug 02 10:06:29.540656 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:29.540598    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1a1c7b2-df89-4f22-a7b7-1d0edba6b351" containerName="payload-pre-processing"
Aug 02 10:06:29.540656 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:29.540633    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ef0920c-d048-4206-9f41-82e3e6452723" containerName="payload-processing"
Aug 02 10:06:29.540656 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:29.540642    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ef0920c-d048-4206-9f41-82e3e6452723" containerName="payload-processing"
Aug 02 10:06:29.540656 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:29.540655    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7accaf2f-b1a9-464d-ba91-493519b79c73" containerName="maas-api"
Aug 02 10:06:29.540925 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:29.540664    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7accaf2f-b1a9-464d-ba91-493519b79c73" containerName="maas-api"
Aug 02 10:06:29.540925 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:29.540729    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ef0920c-d048-4206-9f41-82e3e6452723" containerName="payload-processing"
Aug 02 10:06:29.540925 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:29.540741    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7accaf2f-b1a9-464d-ba91-493519b79c73" containerName="maas-api"
Aug 02 10:06:29.540925 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:29.540749    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1a1c7b2-df89-4f22-a7b7-1d0edba6b351" containerName="payload-pre-processing"
Aug 02 10:06:29.543915 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:29.543896    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-756eaa56-fb6cf6795-nsnjb"
Aug 02 10:06:29.547246 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:29.547209    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-756eaa56-serving-cert\""
Aug 02 10:06:29.556470 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:29.556444    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-756eaa56-fb6cf6795-nsnjb"]
Aug 02 10:06:29.586774 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:29.586743    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:06:29.709094 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:29.709056    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljmrt\" (UniqueName: \"kubernetes.io/projected/8b888b3f-3572-4fde-aecc-3dab4d8b9987-kube-api-access-ljmrt\") pod \"maas-api-e2e-mt-756eaa56-fb6cf6795-nsnjb\" (UID: \"8b888b3f-3572-4fde-aecc-3dab4d8b9987\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-756eaa56-fb6cf6795-nsnjb"
Aug 02 10:06:29.709290 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:29.709127    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b888b3f-3572-4fde-aecc-3dab4d8b9987-maas-api-tls\") pod \"maas-api-e2e-mt-756eaa56-fb6cf6795-nsnjb\" (UID: \"8b888b3f-3572-4fde-aecc-3dab4d8b9987\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-756eaa56-fb6cf6795-nsnjb"
Aug 02 10:06:29.709290 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:29.709154    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b888b3f-3572-4fde-aecc-3dab4d8b9987-openshift-service-ca\") pod \"maas-api-e2e-mt-756eaa56-fb6cf6795-nsnjb\" (UID: \"8b888b3f-3572-4fde-aecc-3dab4d8b9987\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-756eaa56-fb6cf6795-nsnjb"
Aug 02 10:06:29.809803 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:29.809716    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljmrt\" (UniqueName: \"kubernetes.io/projected/8b888b3f-3572-4fde-aecc-3dab4d8b9987-kube-api-access-ljmrt\") pod \"maas-api-e2e-mt-756eaa56-fb6cf6795-nsnjb\" (UID: \"8b888b3f-3572-4fde-aecc-3dab4d8b9987\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-756eaa56-fb6cf6795-nsnjb"
Aug 02 10:06:29.809803 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:29.809772    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b888b3f-3572-4fde-aecc-3dab4d8b9987-maas-api-tls\") pod \"maas-api-e2e-mt-756eaa56-fb6cf6795-nsnjb\" (UID: \"8b888b3f-3572-4fde-aecc-3dab4d8b9987\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-756eaa56-fb6cf6795-nsnjb"
Aug 02 10:06:29.809803 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:29.809797    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b888b3f-3572-4fde-aecc-3dab4d8b9987-openshift-service-ca\") pod \"maas-api-e2e-mt-756eaa56-fb6cf6795-nsnjb\" (UID: \"8b888b3f-3572-4fde-aecc-3dab4d8b9987\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-756eaa56-fb6cf6795-nsnjb"
Aug 02 10:06:29.810085 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:06:29.809932    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-756eaa56-serving-cert: secret "maas-api-e2e-mt-756eaa56-serving-cert" not found
Aug 02 10:06:29.810085 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:06:29.810007    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b888b3f-3572-4fde-aecc-3dab4d8b9987-maas-api-tls podName:8b888b3f-3572-4fde-aecc-3dab4d8b9987 nodeName:}" failed. No retries permitted until 2026-08-02 10:06:30.309990283 +0000 UTC m=+2014.024795725 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8b888b3f-3572-4fde-aecc-3dab4d8b9987-maas-api-tls") pod "maas-api-e2e-mt-756eaa56-fb6cf6795-nsnjb" (UID: "8b888b3f-3572-4fde-aecc-3dab4d8b9987") : secret "maas-api-e2e-mt-756eaa56-serving-cert" not found
Aug 02 10:06:29.810539 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:29.810518    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b888b3f-3572-4fde-aecc-3dab4d8b9987-openshift-service-ca\") pod \"maas-api-e2e-mt-756eaa56-fb6cf6795-nsnjb\" (UID: \"8b888b3f-3572-4fde-aecc-3dab4d8b9987\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-756eaa56-fb6cf6795-nsnjb"
Aug 02 10:06:29.821018 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:29.820997    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljmrt\" (UniqueName: \"kubernetes.io/projected/8b888b3f-3572-4fde-aecc-3dab4d8b9987-kube-api-access-ljmrt\") pod \"maas-api-e2e-mt-756eaa56-fb6cf6795-nsnjb\" (UID: \"8b888b3f-3572-4fde-aecc-3dab4d8b9987\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-756eaa56-fb6cf6795-nsnjb"
Aug 02 10:06:30.064403 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:30.064321    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:06:30.167759 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:30.167726    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:06:30.172618 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:30.172597    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-756eaa56-67d96467b8-b4xcj"]
Aug 02 10:06:30.179648 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:30.179618    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-756eaa56-67d96467b8-b4xcj"
Aug 02 10:06:30.181991 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:30.181969    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-756eaa56\""
Aug 02 10:06:30.185451 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:30.185430    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-756eaa56-67d96467b8-b4xcj"]
Aug 02 10:06:30.197894 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:30.197868    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-756eaa56-66876958b-m6g6v"]
Aug 02 10:06:30.201073 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:30.201054    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-756eaa56-66876958b-m6g6v"
Aug 02 10:06:30.203221 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:30.203198    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-756eaa56-dockercfg-q9rl5\""
Aug 02 10:06:30.209638 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:30.209614    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-756eaa56-66876958b-m6g6v"]
Aug 02 10:06:30.314565 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:30.314470    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85ba8807-fdd2-4b0e-a402-f7114cedeb40-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-756eaa56-67d96467b8-b4xcj\" (UID: \"85ba8807-fdd2-4b0e-a402-f7114cedeb40\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-756eaa56-67d96467b8-b4xcj"
Aug 02 10:06:30.314730 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:30.314574    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znpgw\" (UniqueName: \"kubernetes.io/projected/85ba8807-fdd2-4b0e-a402-f7114cedeb40-kube-api-access-znpgw\") pod \"payload-pre-processing-e2e-mt-756eaa56-67d96467b8-b4xcj\" (UID: \"85ba8807-fdd2-4b0e-a402-f7114cedeb40\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-756eaa56-67d96467b8-b4xcj"
Aug 02 10:06:30.314730 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:30.314650    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b888b3f-3572-4fde-aecc-3dab4d8b9987-maas-api-tls\") pod \"maas-api-e2e-mt-756eaa56-fb6cf6795-nsnjb\" (UID: \"8b888b3f-3572-4fde-aecc-3dab4d8b9987\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-756eaa56-fb6cf6795-nsnjb"
Aug 02 10:06:30.314730 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:30.314689    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b658e0fc-2dae-4342-acab-d9a9fc884407-plugins-config-volume\") pod \"payload-processing-e2e-mt-756eaa56-66876958b-m6g6v\" (UID: \"b658e0fc-2dae-4342-acab-d9a9fc884407\") " pod="openshift-ingress/payload-processing-e2e-mt-756eaa56-66876958b-m6g6v"
Aug 02 10:06:30.314842 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:30.314763    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcccb\" (UniqueName: \"kubernetes.io/projected/b658e0fc-2dae-4342-acab-d9a9fc884407-kube-api-access-jcccb\") pod \"payload-processing-e2e-mt-756eaa56-66876958b-m6g6v\" (UID: \"b658e0fc-2dae-4342-acab-d9a9fc884407\") " pod="openshift-ingress/payload-processing-e2e-mt-756eaa56-66876958b-m6g6v"
Aug 02 10:06:30.317051 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:30.317034    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b888b3f-3572-4fde-aecc-3dab4d8b9987-maas-api-tls\") pod \"maas-api-e2e-mt-756eaa56-fb6cf6795-nsnjb\" (UID: \"8b888b3f-3572-4fde-aecc-3dab4d8b9987\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-756eaa56-fb6cf6795-nsnjb"
Aug 02 10:06:30.415951 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:30.415909    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcccb\" (UniqueName: \"kubernetes.io/projected/b658e0fc-2dae-4342-acab-d9a9fc884407-kube-api-access-jcccb\") pod \"payload-processing-e2e-mt-756eaa56-66876958b-m6g6v\" (UID: \"b658e0fc-2dae-4342-acab-d9a9fc884407\") " pod="openshift-ingress/payload-processing-e2e-mt-756eaa56-66876958b-m6g6v"
Aug 02 10:06:30.415951 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:30.415960    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85ba8807-fdd2-4b0e-a402-f7114cedeb40-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-756eaa56-67d96467b8-b4xcj\" (UID: \"85ba8807-fdd2-4b0e-a402-f7114cedeb40\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-756eaa56-67d96467b8-b4xcj"
Aug 02 10:06:30.416157 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:30.416001    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znpgw\" (UniqueName: \"kubernetes.io/projected/85ba8807-fdd2-4b0e-a402-f7114cedeb40-kube-api-access-znpgw\") pod \"payload-pre-processing-e2e-mt-756eaa56-67d96467b8-b4xcj\" (UID: \"85ba8807-fdd2-4b0e-a402-f7114cedeb40\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-756eaa56-67d96467b8-b4xcj"
Aug 02 10:06:30.416157 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:30.416023    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b658e0fc-2dae-4342-acab-d9a9fc884407-plugins-config-volume\") pod \"payload-processing-e2e-mt-756eaa56-66876958b-m6g6v\" (UID: \"b658e0fc-2dae-4342-acab-d9a9fc884407\") " pod="openshift-ingress/payload-processing-e2e-mt-756eaa56-66876958b-m6g6v"
Aug 02 10:06:30.416724 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:30.416706    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b658e0fc-2dae-4342-acab-d9a9fc884407-plugins-config-volume\") pod \"payload-processing-e2e-mt-756eaa56-66876958b-m6g6v\" (UID: \"b658e0fc-2dae-4342-acab-d9a9fc884407\") " pod="openshift-ingress/payload-processing-e2e-mt-756eaa56-66876958b-m6g6v"
Aug 02 10:06:30.416768 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:30.416706    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85ba8807-fdd2-4b0e-a402-f7114cedeb40-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-756eaa56-67d96467b8-b4xcj\" (UID: \"85ba8807-fdd2-4b0e-a402-f7114cedeb40\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-756eaa56-67d96467b8-b4xcj"
Aug 02 10:06:30.423373 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:30.423346    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znpgw\" (UniqueName: \"kubernetes.io/projected/85ba8807-fdd2-4b0e-a402-f7114cedeb40-kube-api-access-znpgw\") pod \"payload-pre-processing-e2e-mt-756eaa56-67d96467b8-b4xcj\" (UID: \"85ba8807-fdd2-4b0e-a402-f7114cedeb40\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-756eaa56-67d96467b8-b4xcj"
Aug 02 10:06:30.423490 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:30.423426    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcccb\" (UniqueName: \"kubernetes.io/projected/b658e0fc-2dae-4342-acab-d9a9fc884407-kube-api-access-jcccb\") pod \"payload-processing-e2e-mt-756eaa56-66876958b-m6g6v\" (UID: \"b658e0fc-2dae-4342-acab-d9a9fc884407\") " pod="openshift-ingress/payload-processing-e2e-mt-756eaa56-66876958b-m6g6v"
Aug 02 10:06:30.453875 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:30.453839    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-756eaa56-fb6cf6795-nsnjb"
Aug 02 10:06:30.491907 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:30.491871    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-756eaa56-67d96467b8-b4xcj"
Aug 02 10:06:30.512466 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:30.512298    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-756eaa56-66876958b-m6g6v"
Aug 02 10:06:30.598955 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:30.598925    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-756eaa56-fb6cf6795-nsnjb"]
Aug 02 10:06:30.600962 ip-10-0-141-232 kubenswrapper[2564]: W0802 10:06:30.600938    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b888b3f_3572_4fde_aecc_3dab4d8b9987.slice/crio-7a9e5207c0b8f6486ab231eee6211e2ee311888f824dff0b4f5c85dbbac50b16 WatchSource:0}: Error finding container 7a9e5207c0b8f6486ab231eee6211e2ee311888f824dff0b4f5c85dbbac50b16: Status 404 returned error can't find the container with id 7a9e5207c0b8f6486ab231eee6211e2ee311888f824dff0b4f5c85dbbac50b16
Aug 02 10:06:30.644701 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:30.644673    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-756eaa56-67d96467b8-b4xcj"]
Aug 02 10:06:30.647758 ip-10-0-141-232 kubenswrapper[2564]: W0802 10:06:30.647734    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85ba8807_fdd2_4b0e_a402_f7114cedeb40.slice/crio-577efea2a6ec64572ad7bcfefa4b57089eedcd74fcd4bed7bcdce023e496b9e3 WatchSource:0}: Error finding container 577efea2a6ec64572ad7bcfefa4b57089eedcd74fcd4bed7bcdce023e496b9e3: Status 404 returned error can't find the container with id 577efea2a6ec64572ad7bcfefa4b57089eedcd74fcd4bed7bcdce023e496b9e3
Aug 02 10:06:30.662828 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:30.662801    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-756eaa56-66876958b-m6g6v"]
Aug 02 10:06:30.665647 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:30.665618    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-756eaa56-67d96467b8-b4xcj" event={"ID":"85ba8807-fdd2-4b0e-a402-f7114cedeb40","Type":"ContainerStarted","Data":"577efea2a6ec64572ad7bcfefa4b57089eedcd74fcd4bed7bcdce023e496b9e3"}
Aug 02 10:06:30.665858 ip-10-0-141-232 kubenswrapper[2564]: W0802 10:06:30.665817    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb658e0fc_2dae_4342_acab_d9a9fc884407.slice/crio-e083061adfbda96084cb9010b1d28766431529b7070ce7348734e1b5ba9ef04e WatchSource:0}: Error finding container e083061adfbda96084cb9010b1d28766431529b7070ce7348734e1b5ba9ef04e: Status 404 returned error can't find the container with id e083061adfbda96084cb9010b1d28766431529b7070ce7348734e1b5ba9ef04e
Aug 02 10:06:30.666835 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:30.666808    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-756eaa56-fb6cf6795-nsnjb" event={"ID":"8b888b3f-3572-4fde-aecc-3dab4d8b9987","Type":"ContainerStarted","Data":"7a9e5207c0b8f6486ab231eee6211e2ee311888f824dff0b4f5c85dbbac50b16"}
Aug 02 10:06:31.673275 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:31.673224    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-756eaa56-67d96467b8-b4xcj" event={"ID":"85ba8807-fdd2-4b0e-a402-f7114cedeb40","Type":"ContainerStarted","Data":"1c8e44fc3502471ee989382a148d9a9560c6d3308f8a48ac83fb7f9a4b753ef2"}
Aug 02 10:06:31.673701 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:31.673316    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-756eaa56-67d96467b8-b4xcj"
Aug 02 10:06:31.674771 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:31.674744    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-756eaa56-fb6cf6795-nsnjb" event={"ID":"8b888b3f-3572-4fde-aecc-3dab4d8b9987","Type":"ContainerStarted","Data":"bac52bea262badc1be3083fe6c216c30caf9b6109c302b221bdd5323e2ffa81f"}
Aug 02 10:06:31.674919 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:31.674791    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-756eaa56-fb6cf6795-nsnjb"
Aug 02 10:06:31.676104 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:31.676076    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-756eaa56-66876958b-m6g6v" event={"ID":"b658e0fc-2dae-4342-acab-d9a9fc884407","Type":"ContainerStarted","Data":"e368c841d34b840ea9dd712ee1ba0aaffa11db0a5cdf091e0114b8e6f234eb8d"}
Aug 02 10:06:31.676210 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:31.676123    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-756eaa56-66876958b-m6g6v"
Aug 02 10:06:31.676210 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:31.676141    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-756eaa56-66876958b-m6g6v" event={"ID":"b658e0fc-2dae-4342-acab-d9a9fc884407","Type":"ContainerStarted","Data":"e083061adfbda96084cb9010b1d28766431529b7070ce7348734e1b5ba9ef04e"}
Aug 02 10:06:31.688735 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:31.688696    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-756eaa56-67d96467b8-b4xcj" podStartSLOduration=1.688682167 podStartE2EDuration="1.688682167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:06:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:06:31.687413568 +0000 UTC m=+2015.402219017" watchObservedRunningTime="2026-08-02 10:06:31.688682167 +0000 UTC m=+2015.403488012"
Aug 02 10:06:31.703579 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:31.703540    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-756eaa56-fb6cf6795-nsnjb" podStartSLOduration=1.9832816549999999 podStartE2EDuration="2.703527114s" totalImagesPullingTime="720.245459ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:06:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:06:31.702131617 +0000 UTC m=+2015.416937065" watchObservedRunningTime="2026-08-02 10:06:31.703527114 +0000 UTC m=+2015.418332562"
Aug 02 10:06:31.718213 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:31.718174    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-756eaa56-66876958b-m6g6v" podStartSLOduration=1.718157072 podStartE2EDuration="1.718157072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:06:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:06:31.717060595 +0000 UTC m=+2015.431866045" watchObservedRunningTime="2026-08-02 10:06:31.718157072 +0000 UTC m=+2015.432962516"
Aug 02 10:06:37.686111 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:37.686084    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-756eaa56-fb6cf6795-nsnjb"
Aug 02 10:06:42.682938 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:42.682909    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-756eaa56-66876958b-m6g6v"
Aug 02 10:06:42.683514 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:42.682987    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-756eaa56-67d96467b8-b4xcj"
Aug 02 10:06:53.459465 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:53.459430    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:06:54.126341 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:54.126311    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:06:54.224090 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:54.224059    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:06:54.571730 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:06:54.571646    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:07:00.229770 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:00.229733    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:07:00.334977 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:00.334942    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:07:11.061584 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.061550    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-756eaa56-fb6cf6795-nsnjb"]
Aug 02 10:07:11.062126 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.061818    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-756eaa56-fb6cf6795-nsnjb" podUID="8b888b3f-3572-4fde-aecc-3dab4d8b9987" containerName="maas-api" containerID="cri-o://bac52bea262badc1be3083fe6c216c30caf9b6109c302b221bdd5323e2ffa81f" gracePeriod=30
Aug 02 10:07:11.086881 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.086857    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:07:11.118021 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.117983    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-756eaa56-66876958b-m6g6v"]
Aug 02 10:07:11.118327 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.118267    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-756eaa56-66876958b-m6g6v" podUID="b658e0fc-2dae-4342-acab-d9a9fc884407" containerName="payload-processing" containerID="cri-o://e368c841d34b840ea9dd712ee1ba0aaffa11db0a5cdf091e0114b8e6f234eb8d" gracePeriod=30
Aug 02 10:07:11.123581 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.123339    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-756eaa56-67d96467b8-b4xcj"]
Aug 02 10:07:11.124413 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.123979    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-756eaa56-67d96467b8-b4xcj" podUID="85ba8807-fdd2-4b0e-a402-f7114cedeb40" containerName="payload-pre-processing" containerID="cri-o://1c8e44fc3502471ee989382a148d9a9560c6d3308f8a48ac83fb7f9a4b753ef2" gracePeriod=30
Aug 02 10:07:11.155982 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.155952    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:07:11.216729 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.216697    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:07:11.317682 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.317610    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:07:11.394293 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.394220    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-756eaa56-66876958b-m6g6v"
Aug 02 10:07:11.397370 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.397348    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-756eaa56-fb6cf6795-nsnjb"
Aug 02 10:07:11.422476 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.422455    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-756eaa56-67d96467b8-b4xcj"
Aug 02 10:07:11.532017 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.531984    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b888b3f-3572-4fde-aecc-3dab4d8b9987-maas-api-tls\") pod \"8b888b3f-3572-4fde-aecc-3dab4d8b9987\" (UID: \"8b888b3f-3572-4fde-aecc-3dab4d8b9987\") "
Aug 02 10:07:11.532209 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.532026    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljmrt\" (UniqueName: \"kubernetes.io/projected/8b888b3f-3572-4fde-aecc-3dab4d8b9987-kube-api-access-ljmrt\") pod \"8b888b3f-3572-4fde-aecc-3dab4d8b9987\" (UID: \"8b888b3f-3572-4fde-aecc-3dab4d8b9987\") "
Aug 02 10:07:11.532209 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.532051    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85ba8807-fdd2-4b0e-a402-f7114cedeb40-plugins-config-volume\") pod \"85ba8807-fdd2-4b0e-a402-f7114cedeb40\" (UID: \"85ba8807-fdd2-4b0e-a402-f7114cedeb40\") "
Aug 02 10:07:11.532209 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.532067    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znpgw\" (UniqueName: \"kubernetes.io/projected/85ba8807-fdd2-4b0e-a402-f7114cedeb40-kube-api-access-znpgw\") pod \"85ba8807-fdd2-4b0e-a402-f7114cedeb40\" (UID: \"85ba8807-fdd2-4b0e-a402-f7114cedeb40\") "
Aug 02 10:07:11.532209 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.532087    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jcccb\" (UniqueName: \"kubernetes.io/projected/b658e0fc-2dae-4342-acab-d9a9fc884407-kube-api-access-jcccb\") pod \"b658e0fc-2dae-4342-acab-d9a9fc884407\" (UID: \"b658e0fc-2dae-4342-acab-d9a9fc884407\") "
Aug 02 10:07:11.532209 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.532162    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b658e0fc-2dae-4342-acab-d9a9fc884407-plugins-config-volume\") pod \"b658e0fc-2dae-4342-acab-d9a9fc884407\" (UID: \"b658e0fc-2dae-4342-acab-d9a9fc884407\") "
Aug 02 10:07:11.532542 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.532228    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b888b3f-3572-4fde-aecc-3dab4d8b9987-openshift-service-ca\") pod \"8b888b3f-3572-4fde-aecc-3dab4d8b9987\" (UID: \"8b888b3f-3572-4fde-aecc-3dab4d8b9987\") "
Aug 02 10:07:11.532542 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.532505    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85ba8807-fdd2-4b0e-a402-f7114cedeb40-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "85ba8807-fdd2-4b0e-a402-f7114cedeb40" (UID: "85ba8807-fdd2-4b0e-a402-f7114cedeb40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 02 10:07:11.532542 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.532534    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b658e0fc-2dae-4342-acab-d9a9fc884407-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b658e0fc-2dae-4342-acab-d9a9fc884407" (UID: "b658e0fc-2dae-4342-acab-d9a9fc884407"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 02 10:07:11.532711 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.532619    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b888b3f-3572-4fde-aecc-3dab4d8b9987-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8b888b3f-3572-4fde-aecc-3dab4d8b9987" (UID: "8b888b3f-3572-4fde-aecc-3dab4d8b9987"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 02 10:07:11.534275 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.534247    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b888b3f-3572-4fde-aecc-3dab4d8b9987-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8b888b3f-3572-4fde-aecc-3dab4d8b9987" (UID: "8b888b3f-3572-4fde-aecc-3dab4d8b9987"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 02 10:07:11.534388 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.534335    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b658e0fc-2dae-4342-acab-d9a9fc884407-kube-api-access-jcccb" (OuterVolumeSpecName: "kube-api-access-jcccb") pod "b658e0fc-2dae-4342-acab-d9a9fc884407" (UID: "b658e0fc-2dae-4342-acab-d9a9fc884407"). InnerVolumeSpecName "kube-api-access-jcccb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 02 10:07:11.534388 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.534358    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b888b3f-3572-4fde-aecc-3dab4d8b9987-kube-api-access-ljmrt" (OuterVolumeSpecName: "kube-api-access-ljmrt") pod "8b888b3f-3572-4fde-aecc-3dab4d8b9987" (UID: "8b888b3f-3572-4fde-aecc-3dab4d8b9987"). InnerVolumeSpecName "kube-api-access-ljmrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 02 10:07:11.534471 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.534415    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85ba8807-fdd2-4b0e-a402-f7114cedeb40-kube-api-access-znpgw" (OuterVolumeSpecName: "kube-api-access-znpgw") pod "85ba8807-fdd2-4b0e-a402-f7114cedeb40" (UID: "85ba8807-fdd2-4b0e-a402-f7114cedeb40"). InnerVolumeSpecName "kube-api-access-znpgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 02 10:07:11.633214 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.633182    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b888b3f-3572-4fde-aecc-3dab4d8b9987-maas-api-tls\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:07:11.633214 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.633212    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljmrt\" (UniqueName: \"kubernetes.io/projected/8b888b3f-3572-4fde-aecc-3dab4d8b9987-kube-api-access-ljmrt\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:07:11.633417 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.633225    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85ba8807-fdd2-4b0e-a402-f7114cedeb40-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:07:11.633417 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.633252    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znpgw\" (UniqueName: \"kubernetes.io/projected/85ba8807-fdd2-4b0e-a402-f7114cedeb40-kube-api-access-znpgw\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:07:11.633417 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.633261    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcccb\" (UniqueName: \"kubernetes.io/projected/b658e0fc-2dae-4342-acab-d9a9fc884407-kube-api-access-jcccb\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:07:11.633417 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.633269    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b658e0fc-2dae-4342-acab-d9a9fc884407-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:07:11.633417 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.633277    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b888b3f-3572-4fde-aecc-3dab4d8b9987-openshift-service-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:07:11.637692 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.637666    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:07:11.840985 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.840952    2564 generic.go:356] "Generic (PLEG): container finished" podID="b658e0fc-2dae-4342-acab-d9a9fc884407" containerID="e368c841d34b840ea9dd712ee1ba0aaffa11db0a5cdf091e0114b8e6f234eb8d" exitCode=0
Aug 02 10:07:11.841182 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.840993    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-756eaa56-66876958b-m6g6v"
Aug 02 10:07:11.841182 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.841038    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-756eaa56-66876958b-m6g6v" event={"ID":"b658e0fc-2dae-4342-acab-d9a9fc884407","Type":"ContainerDied","Data":"e368c841d34b840ea9dd712ee1ba0aaffa11db0a5cdf091e0114b8e6f234eb8d"}
Aug 02 10:07:11.841182 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.841093    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-756eaa56-66876958b-m6g6v" event={"ID":"b658e0fc-2dae-4342-acab-d9a9fc884407","Type":"ContainerDied","Data":"e083061adfbda96084cb9010b1d28766431529b7070ce7348734e1b5ba9ef04e"}
Aug 02 10:07:11.841182 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.841124    2564 scope.go:117] "RemoveContainer" containerID="e368c841d34b840ea9dd712ee1ba0aaffa11db0a5cdf091e0114b8e6f234eb8d"
Aug 02 10:07:11.842411 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.842379    2564 generic.go:356] "Generic (PLEG): container finished" podID="85ba8807-fdd2-4b0e-a402-f7114cedeb40" containerID="1c8e44fc3502471ee989382a148d9a9560c6d3308f8a48ac83fb7f9a4b753ef2" exitCode=0
Aug 02 10:07:11.842506 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.842422    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-756eaa56-67d96467b8-b4xcj"
Aug 02 10:07:11.842506 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.842429    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-756eaa56-67d96467b8-b4xcj" event={"ID":"85ba8807-fdd2-4b0e-a402-f7114cedeb40","Type":"ContainerDied","Data":"1c8e44fc3502471ee989382a148d9a9560c6d3308f8a48ac83fb7f9a4b753ef2"}
Aug 02 10:07:11.842506 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.842491    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-756eaa56-67d96467b8-b4xcj" event={"ID":"85ba8807-fdd2-4b0e-a402-f7114cedeb40","Type":"ContainerDied","Data":"577efea2a6ec64572ad7bcfefa4b57089eedcd74fcd4bed7bcdce023e496b9e3"}
Aug 02 10:07:11.843815 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.843792    2564 generic.go:356] "Generic (PLEG): container finished" podID="8b888b3f-3572-4fde-aecc-3dab4d8b9987" containerID="bac52bea262badc1be3083fe6c216c30caf9b6109c302b221bdd5323e2ffa81f" exitCode=0
Aug 02 10:07:11.843815 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.843820    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-756eaa56-fb6cf6795-nsnjb"
Aug 02 10:07:11.844011 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.843834    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-756eaa56-fb6cf6795-nsnjb" event={"ID":"8b888b3f-3572-4fde-aecc-3dab4d8b9987","Type":"ContainerDied","Data":"bac52bea262badc1be3083fe6c216c30caf9b6109c302b221bdd5323e2ffa81f"}
Aug 02 10:07:11.844011 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.843893    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-756eaa56-fb6cf6795-nsnjb" event={"ID":"8b888b3f-3572-4fde-aecc-3dab4d8b9987","Type":"ContainerDied","Data":"7a9e5207c0b8f6486ab231eee6211e2ee311888f824dff0b4f5c85dbbac50b16"}
Aug 02 10:07:11.850666 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.850638    2564 scope.go:117] "RemoveContainer" containerID="e368c841d34b840ea9dd712ee1ba0aaffa11db0a5cdf091e0114b8e6f234eb8d"
Aug 02 10:07:11.850949 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:07:11.850922    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e368c841d34b840ea9dd712ee1ba0aaffa11db0a5cdf091e0114b8e6f234eb8d\": container with ID starting with e368c841d34b840ea9dd712ee1ba0aaffa11db0a5cdf091e0114b8e6f234eb8d not found: ID does not exist" containerID="e368c841d34b840ea9dd712ee1ba0aaffa11db0a5cdf091e0114b8e6f234eb8d"
Aug 02 10:07:11.851047 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.850959    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e368c841d34b840ea9dd712ee1ba0aaffa11db0a5cdf091e0114b8e6f234eb8d"} err="failed to get container status \"e368c841d34b840ea9dd712ee1ba0aaffa11db0a5cdf091e0114b8e6f234eb8d\": rpc error: code = NotFound desc = could not find container \"e368c841d34b840ea9dd712ee1ba0aaffa11db0a5cdf091e0114b8e6f234eb8d\": container with ID starting with e368c841d34b840ea9dd712ee1ba0aaffa11db0a5cdf091e0114b8e6f234eb8d not found: ID does not exist"
Aug 02 10:07:11.851047 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.850982    2564 scope.go:117] "RemoveContainer" containerID="1c8e44fc3502471ee989382a148d9a9560c6d3308f8a48ac83fb7f9a4b753ef2"
Aug 02 10:07:11.859159 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.859142    2564 scope.go:117] "RemoveContainer" containerID="1c8e44fc3502471ee989382a148d9a9560c6d3308f8a48ac83fb7f9a4b753ef2"
Aug 02 10:07:11.859499 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:07:11.859479    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c8e44fc3502471ee989382a148d9a9560c6d3308f8a48ac83fb7f9a4b753ef2\": container with ID starting with 1c8e44fc3502471ee989382a148d9a9560c6d3308f8a48ac83fb7f9a4b753ef2 not found: ID does not exist" containerID="1c8e44fc3502471ee989382a148d9a9560c6d3308f8a48ac83fb7f9a4b753ef2"
Aug 02 10:07:11.859564 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.859505    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c8e44fc3502471ee989382a148d9a9560c6d3308f8a48ac83fb7f9a4b753ef2"} err="failed to get container status \"1c8e44fc3502471ee989382a148d9a9560c6d3308f8a48ac83fb7f9a4b753ef2\": rpc error: code = NotFound desc = could not find container \"1c8e44fc3502471ee989382a148d9a9560c6d3308f8a48ac83fb7f9a4b753ef2\": container with ID starting with 1c8e44fc3502471ee989382a148d9a9560c6d3308f8a48ac83fb7f9a4b753ef2 not found: ID does not exist"
Aug 02 10:07:11.859564 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.859520    2564 scope.go:117] "RemoveContainer" containerID="bac52bea262badc1be3083fe6c216c30caf9b6109c302b221bdd5323e2ffa81f"
Aug 02 10:07:11.866347 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.866328    2564 scope.go:117] "RemoveContainer" containerID="bac52bea262badc1be3083fe6c216c30caf9b6109c302b221bdd5323e2ffa81f"
Aug 02 10:07:11.866590 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:07:11.866572    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bac52bea262badc1be3083fe6c216c30caf9b6109c302b221bdd5323e2ffa81f\": container with ID starting with bac52bea262badc1be3083fe6c216c30caf9b6109c302b221bdd5323e2ffa81f not found: ID does not exist" containerID="bac52bea262badc1be3083fe6c216c30caf9b6109c302b221bdd5323e2ffa81f"
Aug 02 10:07:11.866644 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.866594    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bac52bea262badc1be3083fe6c216c30caf9b6109c302b221bdd5323e2ffa81f"} err="failed to get container status \"bac52bea262badc1be3083fe6c216c30caf9b6109c302b221bdd5323e2ffa81f\": rpc error: code = NotFound desc = could not find container \"bac52bea262badc1be3083fe6c216c30caf9b6109c302b221bdd5323e2ffa81f\": container with ID starting with bac52bea262badc1be3083fe6c216c30caf9b6109c302b221bdd5323e2ffa81f not found: ID does not exist"
Aug 02 10:07:11.867018 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.867001    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-756eaa56-67d96467b8-b4xcj"]
Aug 02 10:07:11.870254 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.870215    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-756eaa56-67d96467b8-b4xcj"]
Aug 02 10:07:11.879021 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.879001    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-756eaa56-66876958b-m6g6v"]
Aug 02 10:07:11.882919 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.882900    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-756eaa56-66876958b-m6g6v"]
Aug 02 10:07:11.892431 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.892383    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-756eaa56-fb6cf6795-nsnjb"]
Aug 02 10:07:11.895851 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:11.895833    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-756eaa56-fb6cf6795-nsnjb"]
Aug 02 10:07:12.883915 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:12.883881    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85ba8807-fdd2-4b0e-a402-f7114cedeb40" path="/var/lib/kubelet/pods/85ba8807-fdd2-4b0e-a402-f7114cedeb40/volumes"
Aug 02 10:07:12.884432 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:12.884412    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b888b3f-3572-4fde-aecc-3dab4d8b9987" path="/var/lib/kubelet/pods/8b888b3f-3572-4fde-aecc-3dab4d8b9987/volumes"
Aug 02 10:07:12.884830 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:12.884808    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b658e0fc-2dae-4342-acab-d9a9fc884407" path="/var/lib/kubelet/pods/b658e0fc-2dae-4342-acab-d9a9fc884407/volumes"
Aug 02 10:07:17.486504 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:17.486472    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:07:17.586989 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:17.586959    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:07:17.759457 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:17.759365    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:07:17.974630 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:17.974599    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:07:18.120786 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:18.120681    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:07:18.216785 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:18.216730    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:07:18.315676 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:18.315643    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:07:23.881425 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:23.881352    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:07:23.984660 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:23.984624    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:07:24.090666 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:24.090634    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:07:24.229158 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:24.229083    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:07:29.905279 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:29.905248    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:07:31.148992 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:07:31.148963    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:09:32.911860 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:32.911829    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:09:33.194832 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:33.194743    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:09:33.496127 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:33.496031    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:09:36.391444 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:36.391376    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:09:36.484883 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:36.484847    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:09:37.465973 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:37.465942    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7e5ef7e7-64b4c9f6fb-kbsxs"]
Aug 02 10:09:37.466385 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:37.466249    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b888b3f-3572-4fde-aecc-3dab4d8b9987" containerName="maas-api"
Aug 02 10:09:37.466385 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:37.466261    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b888b3f-3572-4fde-aecc-3dab4d8b9987" containerName="maas-api"
Aug 02 10:09:37.466385 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:37.466270    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b658e0fc-2dae-4342-acab-d9a9fc884407" containerName="payload-processing"
Aug 02 10:09:37.466385 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:37.466276    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b658e0fc-2dae-4342-acab-d9a9fc884407" containerName="payload-processing"
Aug 02 10:09:37.466385 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:37.466284    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85ba8807-fdd2-4b0e-a402-f7114cedeb40" containerName="payload-pre-processing"
Aug 02 10:09:37.466385 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:37.466290    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="85ba8807-fdd2-4b0e-a402-f7114cedeb40" containerName="payload-pre-processing"
Aug 02 10:09:37.466385 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:37.466341    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="85ba8807-fdd2-4b0e-a402-f7114cedeb40" containerName="payload-pre-processing"
Aug 02 10:09:37.466385 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:37.466348    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b888b3f-3572-4fde-aecc-3dab4d8b9987" containerName="maas-api"
Aug 02 10:09:37.466385 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:37.466356    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b658e0fc-2dae-4342-acab-d9a9fc884407" containerName="payload-processing"
Aug 02 10:09:37.469117 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:37.469094    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e5ef7e7-64b4c9f6fb-kbsxs"
Aug 02 10:09:37.471480 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:37.471459    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-7e5ef7e7-serving-cert\""
Aug 02 10:09:37.479600 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:37.479577    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7e5ef7e7-64b4c9f6fb-kbsxs"]
Aug 02 10:09:37.498874 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:37.498843    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:09:37.541903 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:37.541880    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbmsf\" (UniqueName: \"kubernetes.io/projected/ed67adaa-842e-4f96-bf01-d314c6d77ebe-kube-api-access-wbmsf\") pod \"maas-api-e2e-mt-7e5ef7e7-64b4c9f6fb-kbsxs\" (UID: \"ed67adaa-842e-4f96-bf01-d314c6d77ebe\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e5ef7e7-64b4c9f6fb-kbsxs"
Aug 02 10:09:37.542023 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:37.541910    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed67adaa-842e-4f96-bf01-d314c6d77ebe-openshift-service-ca\") pod \"maas-api-e2e-mt-7e5ef7e7-64b4c9f6fb-kbsxs\" (UID: \"ed67adaa-842e-4f96-bf01-d314c6d77ebe\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e5ef7e7-64b4c9f6fb-kbsxs"
Aug 02 10:09:37.542023 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:37.541943    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed67adaa-842e-4f96-bf01-d314c6d77ebe-maas-api-tls\") pod \"maas-api-e2e-mt-7e5ef7e7-64b4c9f6fb-kbsxs\" (UID: \"ed67adaa-842e-4f96-bf01-d314c6d77ebe\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e5ef7e7-64b4c9f6fb-kbsxs"
Aug 02 10:09:37.642436 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:37.642411    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbmsf\" (UniqueName: \"kubernetes.io/projected/ed67adaa-842e-4f96-bf01-d314c6d77ebe-kube-api-access-wbmsf\") pod \"maas-api-e2e-mt-7e5ef7e7-64b4c9f6fb-kbsxs\" (UID: \"ed67adaa-842e-4f96-bf01-d314c6d77ebe\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e5ef7e7-64b4c9f6fb-kbsxs"
Aug 02 10:09:37.642436 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:37.642442    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed67adaa-842e-4f96-bf01-d314c6d77ebe-openshift-service-ca\") pod \"maas-api-e2e-mt-7e5ef7e7-64b4c9f6fb-kbsxs\" (UID: \"ed67adaa-842e-4f96-bf01-d314c6d77ebe\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e5ef7e7-64b4c9f6fb-kbsxs"
Aug 02 10:09:37.642661 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:37.642470    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed67adaa-842e-4f96-bf01-d314c6d77ebe-maas-api-tls\") pod \"maas-api-e2e-mt-7e5ef7e7-64b4c9f6fb-kbsxs\" (UID: \"ed67adaa-842e-4f96-bf01-d314c6d77ebe\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e5ef7e7-64b4c9f6fb-kbsxs"
Aug 02 10:09:37.642661 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:09:37.642624    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-7e5ef7e7-serving-cert: secret "maas-api-e2e-mt-7e5ef7e7-serving-cert" not found
Aug 02 10:09:37.642771 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:09:37.642704    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed67adaa-842e-4f96-bf01-d314c6d77ebe-maas-api-tls podName:ed67adaa-842e-4f96-bf01-d314c6d77ebe nodeName:}" failed. No retries permitted until 2026-08-02 10:09:38.142682479 +0000 UTC m=+2201.857487913 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ed67adaa-842e-4f96-bf01-d314c6d77ebe-maas-api-tls") pod "maas-api-e2e-mt-7e5ef7e7-64b4c9f6fb-kbsxs" (UID: "ed67adaa-842e-4f96-bf01-d314c6d77ebe") : secret "maas-api-e2e-mt-7e5ef7e7-serving-cert" not found
Aug 02 10:09:37.643057 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:37.643039    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed67adaa-842e-4f96-bf01-d314c6d77ebe-openshift-service-ca\") pod \"maas-api-e2e-mt-7e5ef7e7-64b4c9f6fb-kbsxs\" (UID: \"ed67adaa-842e-4f96-bf01-d314c6d77ebe\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e5ef7e7-64b4c9f6fb-kbsxs"
Aug 02 10:09:37.650613 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:37.650594    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbmsf\" (UniqueName: \"kubernetes.io/projected/ed67adaa-842e-4f96-bf01-d314c6d77ebe-kube-api-access-wbmsf\") pod \"maas-api-e2e-mt-7e5ef7e7-64b4c9f6fb-kbsxs\" (UID: \"ed67adaa-842e-4f96-bf01-d314c6d77ebe\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e5ef7e7-64b4c9f6fb-kbsxs"
Aug 02 10:09:37.665632 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:37.665597    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:09:37.843922 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:37.843814    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:09:37.999589 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:37.999557    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:09:38.089607 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:38.089573    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:09:38.093778 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:38.093752    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7e5ef7e7-76985bcbb8-g6s8x"]
Aug 02 10:09:38.096948 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:38.096908    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e5ef7e7-76985bcbb8-g6s8x"
Aug 02 10:09:38.099096 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:38.099080    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-7e5ef7e7-dockercfg-55g8s\""
Aug 02 10:09:38.099400 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:38.099385    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-7e5ef7e7\""
Aug 02 10:09:38.106169 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:38.106145    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7e5ef7e7-76985bcbb8-g6s8x"]
Aug 02 10:09:38.141068 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:38.141044    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7e5ef7e7-69fdb44446-9z5gn"]
Aug 02 10:09:38.144202 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:38.144186    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7e5ef7e7-69fdb44446-9z5gn"
Aug 02 10:09:38.147250 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:38.147221    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed67adaa-842e-4f96-bf01-d314c6d77ebe-maas-api-tls\") pod \"maas-api-e2e-mt-7e5ef7e7-64b4c9f6fb-kbsxs\" (UID: \"ed67adaa-842e-4f96-bf01-d314c6d77ebe\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e5ef7e7-64b4c9f6fb-kbsxs"
Aug 02 10:09:38.149955 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:38.149933    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed67adaa-842e-4f96-bf01-d314c6d77ebe-maas-api-tls\") pod \"maas-api-e2e-mt-7e5ef7e7-64b4c9f6fb-kbsxs\" (UID: \"ed67adaa-842e-4f96-bf01-d314c6d77ebe\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e5ef7e7-64b4c9f6fb-kbsxs"
Aug 02 10:09:38.154818 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:38.154798    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7e5ef7e7-69fdb44446-9z5gn"]
Aug 02 10:09:38.247872 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:38.247847    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fe54b2d-7431-4307-92b9-abec296278c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7e5ef7e7-76985bcbb8-g6s8x\" (UID: \"3fe54b2d-7431-4307-92b9-abec296278c9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7e5ef7e7-76985bcbb8-g6s8x"
Aug 02 10:09:38.248040 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:38.247881    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx2br\" (UniqueName: \"kubernetes.io/projected/3fe54b2d-7431-4307-92b9-abec296278c9-kube-api-access-hx2br\") pod \"payload-pre-processing-e2e-mt-7e5ef7e7-76985bcbb8-g6s8x\" (UID: \"3fe54b2d-7431-4307-92b9-abec296278c9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7e5ef7e7-76985bcbb8-g6s8x"
Aug 02 10:09:38.248040 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:38.247907    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqbff\" (UniqueName: \"kubernetes.io/projected/7f433036-b7e8-4d47-a458-52598d130b31-kube-api-access-jqbff\") pod \"payload-processing-e2e-mt-7e5ef7e7-69fdb44446-9z5gn\" (UID: \"7f433036-b7e8-4d47-a458-52598d130b31\") " pod="openshift-ingress/payload-processing-e2e-mt-7e5ef7e7-69fdb44446-9z5gn"
Aug 02 10:09:38.248040 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:38.248006    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f433036-b7e8-4d47-a458-52598d130b31-plugins-config-volume\") pod \"payload-processing-e2e-mt-7e5ef7e7-69fdb44446-9z5gn\" (UID: \"7f433036-b7e8-4d47-a458-52598d130b31\") " pod="openshift-ingress/payload-processing-e2e-mt-7e5ef7e7-69fdb44446-9z5gn"
Aug 02 10:09:38.349193 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:38.349108    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fe54b2d-7431-4307-92b9-abec296278c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7e5ef7e7-76985bcbb8-g6s8x\" (UID: \"3fe54b2d-7431-4307-92b9-abec296278c9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7e5ef7e7-76985bcbb8-g6s8x"
Aug 02 10:09:38.349193 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:38.349153    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx2br\" (UniqueName: \"kubernetes.io/projected/3fe54b2d-7431-4307-92b9-abec296278c9-kube-api-access-hx2br\") pod \"payload-pre-processing-e2e-mt-7e5ef7e7-76985bcbb8-g6s8x\" (UID: \"3fe54b2d-7431-4307-92b9-abec296278c9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7e5ef7e7-76985bcbb8-g6s8x"
Aug 02 10:09:38.349193 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:38.349179    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqbff\" (UniqueName: \"kubernetes.io/projected/7f433036-b7e8-4d47-a458-52598d130b31-kube-api-access-jqbff\") pod \"payload-processing-e2e-mt-7e5ef7e7-69fdb44446-9z5gn\" (UID: \"7f433036-b7e8-4d47-a458-52598d130b31\") " pod="openshift-ingress/payload-processing-e2e-mt-7e5ef7e7-69fdb44446-9z5gn"
Aug 02 10:09:38.349492 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:38.349206    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f433036-b7e8-4d47-a458-52598d130b31-plugins-config-volume\") pod \"payload-processing-e2e-mt-7e5ef7e7-69fdb44446-9z5gn\" (UID: \"7f433036-b7e8-4d47-a458-52598d130b31\") " pod="openshift-ingress/payload-processing-e2e-mt-7e5ef7e7-69fdb44446-9z5gn"
Aug 02 10:09:38.349995 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:38.349971    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fe54b2d-7431-4307-92b9-abec296278c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7e5ef7e7-76985bcbb8-g6s8x\" (UID: \"3fe54b2d-7431-4307-92b9-abec296278c9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7e5ef7e7-76985bcbb8-g6s8x"
Aug 02 10:09:38.350353 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:38.350329    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f433036-b7e8-4d47-a458-52598d130b31-plugins-config-volume\") pod \"payload-processing-e2e-mt-7e5ef7e7-69fdb44446-9z5gn\" (UID: \"7f433036-b7e8-4d47-a458-52598d130b31\") " pod="openshift-ingress/payload-processing-e2e-mt-7e5ef7e7-69fdb44446-9z5gn"
Aug 02 10:09:38.362684 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:38.362662    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqbff\" (UniqueName: \"kubernetes.io/projected/7f433036-b7e8-4d47-a458-52598d130b31-kube-api-access-jqbff\") pod \"payload-processing-e2e-mt-7e5ef7e7-69fdb44446-9z5gn\" (UID: \"7f433036-b7e8-4d47-a458-52598d130b31\") " pod="openshift-ingress/payload-processing-e2e-mt-7e5ef7e7-69fdb44446-9z5gn"
Aug 02 10:09:38.362786 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:38.362740    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx2br\" (UniqueName: \"kubernetes.io/projected/3fe54b2d-7431-4307-92b9-abec296278c9-kube-api-access-hx2br\") pod \"payload-pre-processing-e2e-mt-7e5ef7e7-76985bcbb8-g6s8x\" (UID: \"3fe54b2d-7431-4307-92b9-abec296278c9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7e5ef7e7-76985bcbb8-g6s8x"
Aug 02 10:09:38.380425 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:38.380396    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e5ef7e7-64b4c9f6fb-kbsxs"
Aug 02 10:09:38.408656 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:38.408631    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e5ef7e7-76985bcbb8-g6s8x"
Aug 02 10:09:38.462913 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:38.462479    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7e5ef7e7-69fdb44446-9z5gn"
Aug 02 10:09:38.543129 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:38.543063    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7e5ef7e7-64b4c9f6fb-kbsxs"]
Aug 02 10:09:38.546447 ip-10-0-141-232 kubenswrapper[2564]: W0802 10:09:38.546423    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded67adaa_842e_4f96_bf01_d314c6d77ebe.slice/crio-cb7dfc68629d028fd61727932454194d58095fac28abe54dc1ae7402b6c6ded7 WatchSource:0}: Error finding container cb7dfc68629d028fd61727932454194d58095fac28abe54dc1ae7402b6c6ded7: Status 404 returned error can't find the container with id cb7dfc68629d028fd61727932454194d58095fac28abe54dc1ae7402b6c6ded7
Aug 02 10:09:38.548338 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:38.548324    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 02 10:09:38.557562 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:38.557539    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7e5ef7e7-76985bcbb8-g6s8x"]
Aug 02 10:09:38.558825 ip-10-0-141-232 kubenswrapper[2564]: W0802 10:09:38.558794    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fe54b2d_7431_4307_92b9_abec296278c9.slice/crio-d1b5c699d817edc3b1a4b1fd7ca4d7148e1ba7d2af41341f8e8ec13d1a9e0ad2 WatchSource:0}: Error finding container d1b5c699d817edc3b1a4b1fd7ca4d7148e1ba7d2af41341f8e8ec13d1a9e0ad2: Status 404 returned error can't find the container with id d1b5c699d817edc3b1a4b1fd7ca4d7148e1ba7d2af41341f8e8ec13d1a9e0ad2
Aug 02 10:09:38.611870 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:38.611845    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7e5ef7e7-69fdb44446-9z5gn"]
Aug 02 10:09:38.613864 ip-10-0-141-232 kubenswrapper[2564]: W0802 10:09:38.613830    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f433036_b7e8_4d47_a458_52598d130b31.slice/crio-01b55661d9b9cb9101ba9b830a9822b485fae80e91f424fc1cb308d5e2ea35c9 WatchSource:0}: Error finding container 01b55661d9b9cb9101ba9b830a9822b485fae80e91f424fc1cb308d5e2ea35c9: Status 404 returned error can't find the container with id 01b55661d9b9cb9101ba9b830a9822b485fae80e91f424fc1cb308d5e2ea35c9
Aug 02 10:09:39.435131 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:39.435094    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e5ef7e7-76985bcbb8-g6s8x" event={"ID":"3fe54b2d-7431-4307-92b9-abec296278c9","Type":"ContainerStarted","Data":"99bc734788e96a8faa3a0eb1a53e527a45092cb328b25149568adc8ee76f3d77"}
Aug 02 10:09:39.435347 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:39.435195    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e5ef7e7-76985bcbb8-g6s8x"
Aug 02 10:09:39.435347 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:39.435219    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e5ef7e7-76985bcbb8-g6s8x" event={"ID":"3fe54b2d-7431-4307-92b9-abec296278c9","Type":"ContainerStarted","Data":"d1b5c699d817edc3b1a4b1fd7ca4d7148e1ba7d2af41341f8e8ec13d1a9e0ad2"}
Aug 02 10:09:39.436769 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:39.436739    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7e5ef7e7-69fdb44446-9z5gn" event={"ID":"7f433036-b7e8-4d47-a458-52598d130b31","Type":"ContainerStarted","Data":"25c615967327ae8687fbac4e87c2787219509f6fc1d060af241c0caee3a05aef"}
Aug 02 10:09:39.436997 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:39.436788    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7e5ef7e7-69fdb44446-9z5gn" event={"ID":"7f433036-b7e8-4d47-a458-52598d130b31","Type":"ContainerStarted","Data":"01b55661d9b9cb9101ba9b830a9822b485fae80e91f424fc1cb308d5e2ea35c9"}
Aug 02 10:09:39.436997 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:39.436832    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-7e5ef7e7-69fdb44446-9z5gn"
Aug 02 10:09:39.438131 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:39.438109    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e5ef7e7-64b4c9f6fb-kbsxs" event={"ID":"ed67adaa-842e-4f96-bf01-d314c6d77ebe","Type":"ContainerStarted","Data":"cc76ac1b3e0baf00414c21979b0e73fa633389feea9907010b4b7143f2128114"}
Aug 02 10:09:39.438214 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:39.438145    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e5ef7e7-64b4c9f6fb-kbsxs" event={"ID":"ed67adaa-842e-4f96-bf01-d314c6d77ebe","Type":"ContainerStarted","Data":"cb7dfc68629d028fd61727932454194d58095fac28abe54dc1ae7402b6c6ded7"}
Aug 02 10:09:39.438277 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:39.438255    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e5ef7e7-64b4c9f6fb-kbsxs"
Aug 02 10:09:39.455427 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:39.455384    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e5ef7e7-76985bcbb8-g6s8x" podStartSLOduration=1.45536848 podStartE2EDuration="1.45536848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:09:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:09:39.453312937 +0000 UTC m=+2203.168118387" watchObservedRunningTime="2026-08-02 10:09:39.45536848 +0000 UTC m=+2203.170173926"
Aug 02 10:09:39.471082 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:39.471046    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e5ef7e7-64b4c9f6fb-kbsxs" podStartSLOduration=1.9773805800000002 podStartE2EDuration="2.471029408s" totalImagesPullingTime="493.648828ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:09:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:09:39.470478627 +0000 UTC m=+2203.185284077" watchObservedRunningTime="2026-08-02 10:09:39.471029408 +0000 UTC m=+2203.185834857"
Aug 02 10:09:39.486158 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:39.486120    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-7e5ef7e7-69fdb44446-9z5gn" podStartSLOduration=1.486104669 podStartE2EDuration="1.486104669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:09:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:09:39.48558148 +0000 UTC m=+2203.200386929" watchObservedRunningTime="2026-08-02 10:09:39.486104669 +0000 UTC m=+2203.200910118"
Aug 02 10:09:45.450792 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:45.450759    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e5ef7e7-64b4c9f6fb-kbsxs"
Aug 02 10:09:50.447149 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:50.447105    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-7e5ef7e7-69fdb44446-9z5gn"
Aug 02 10:09:50.447600 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:09:50.447187    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e5ef7e7-76985bcbb8-g6s8x"
Aug 02 10:10:01.380197 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:10:01.380158    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:10:11.612904 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:10:11.612861    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:12:24.156197 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:12:24.156117    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:14:04.522659 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:04.522620    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7e5ef7e7-64b4c9f6fb-kbsxs"]
Aug 02 10:14:04.523197 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:04.522840    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e5ef7e7-64b4c9f6fb-kbsxs" podUID="ed67adaa-842e-4f96-bf01-d314c6d77ebe" containerName="maas-api" containerID="cri-o://cc76ac1b3e0baf00414c21979b0e73fa633389feea9907010b4b7143f2128114" gracePeriod=30
Aug 02 10:14:04.573060 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:04.573032    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7e5ef7e7-69fdb44446-9z5gn"]
Aug 02 10:14:04.573316 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:04.573289    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-7e5ef7e7-69fdb44446-9z5gn" podUID="7f433036-b7e8-4d47-a458-52598d130b31" containerName="payload-processing" containerID="cri-o://25c615967327ae8687fbac4e87c2787219509f6fc1d060af241c0caee3a05aef" gracePeriod=30
Aug 02 10:14:04.580383 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:04.580355    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7e5ef7e7-76985bcbb8-g6s8x"]
Aug 02 10:14:04.580662 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:04.580623    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e5ef7e7-76985bcbb8-g6s8x" podUID="3fe54b2d-7431-4307-92b9-abec296278c9" containerName="payload-pre-processing" containerID="cri-o://99bc734788e96a8faa3a0eb1a53e527a45092cb328b25149568adc8ee76f3d77" gracePeriod=30
Aug 02 10:14:04.581135 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:04.581110    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:14:04.820932 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:04.820901    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e5ef7e7-64b4c9f6fb-kbsxs"
Aug 02 10:14:04.854072 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:04.854046    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7e5ef7e7-69fdb44446-9z5gn"
Aug 02 10:14:04.864340 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:04.864318    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e5ef7e7-76985bcbb8-g6s8x"
Aug 02 10:14:04.876172 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:04.875890    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed67adaa-842e-4f96-bf01-d314c6d77ebe-maas-api-tls\") pod \"ed67adaa-842e-4f96-bf01-d314c6d77ebe\" (UID: \"ed67adaa-842e-4f96-bf01-d314c6d77ebe\") "
Aug 02 10:14:04.876172 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:04.875957    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hx2br\" (UniqueName: \"kubernetes.io/projected/3fe54b2d-7431-4307-92b9-abec296278c9-kube-api-access-hx2br\") pod \"3fe54b2d-7431-4307-92b9-abec296278c9\" (UID: \"3fe54b2d-7431-4307-92b9-abec296278c9\") "
Aug 02 10:14:04.876172 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:04.875991    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed67adaa-842e-4f96-bf01-d314c6d77ebe-openshift-service-ca\") pod \"ed67adaa-842e-4f96-bf01-d314c6d77ebe\" (UID: \"ed67adaa-842e-4f96-bf01-d314c6d77ebe\") "
Aug 02 10:14:04.876172 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:04.876021    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fe54b2d-7431-4307-92b9-abec296278c9-plugins-config-volume\") pod \"3fe54b2d-7431-4307-92b9-abec296278c9\" (UID: \"3fe54b2d-7431-4307-92b9-abec296278c9\") "
Aug 02 10:14:04.876172 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:04.876066    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f433036-b7e8-4d47-a458-52598d130b31-plugins-config-volume\") pod \"7f433036-b7e8-4d47-a458-52598d130b31\" (UID: \"7f433036-b7e8-4d47-a458-52598d130b31\") "
Aug 02 10:14:04.876172 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:04.876099    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jqbff\" (UniqueName: \"kubernetes.io/projected/7f433036-b7e8-4d47-a458-52598d130b31-kube-api-access-jqbff\") pod \"7f433036-b7e8-4d47-a458-52598d130b31\" (UID: \"7f433036-b7e8-4d47-a458-52598d130b31\") "
Aug 02 10:14:04.876648 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:04.876619    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed67adaa-842e-4f96-bf01-d314c6d77ebe-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ed67adaa-842e-4f96-bf01-d314c6d77ebe" (UID: "ed67adaa-842e-4f96-bf01-d314c6d77ebe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 02 10:14:04.876800 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:04.876767    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f433036-b7e8-4d47-a458-52598d130b31-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7f433036-b7e8-4d47-a458-52598d130b31" (UID: "7f433036-b7e8-4d47-a458-52598d130b31"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 02 10:14:04.876962 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:04.876939    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fe54b2d-7431-4307-92b9-abec296278c9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3fe54b2d-7431-4307-92b9-abec296278c9" (UID: "3fe54b2d-7431-4307-92b9-abec296278c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 02 10:14:04.878559 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:04.878533    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f433036-b7e8-4d47-a458-52598d130b31-kube-api-access-jqbff" (OuterVolumeSpecName: "kube-api-access-jqbff") pod "7f433036-b7e8-4d47-a458-52598d130b31" (UID: "7f433036-b7e8-4d47-a458-52598d130b31"). InnerVolumeSpecName "kube-api-access-jqbff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 02 10:14:04.878797 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:04.878771    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed67adaa-842e-4f96-bf01-d314c6d77ebe-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ed67adaa-842e-4f96-bf01-d314c6d77ebe" (UID: "ed67adaa-842e-4f96-bf01-d314c6d77ebe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 02 10:14:04.879057 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:04.879023    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fe54b2d-7431-4307-92b9-abec296278c9-kube-api-access-hx2br" (OuterVolumeSpecName: "kube-api-access-hx2br") pod "3fe54b2d-7431-4307-92b9-abec296278c9" (UID: "3fe54b2d-7431-4307-92b9-abec296278c9"). InnerVolumeSpecName "kube-api-access-hx2br". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 02 10:14:04.976609 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:04.976576    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wbmsf\" (UniqueName: \"kubernetes.io/projected/ed67adaa-842e-4f96-bf01-d314c6d77ebe-kube-api-access-wbmsf\") pod \"ed67adaa-842e-4f96-bf01-d314c6d77ebe\" (UID: \"ed67adaa-842e-4f96-bf01-d314c6d77ebe\") "
Aug 02 10:14:04.976811 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:04.976787    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx2br\" (UniqueName: \"kubernetes.io/projected/3fe54b2d-7431-4307-92b9-abec296278c9-kube-api-access-hx2br\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:14:04.976811 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:04.976809    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed67adaa-842e-4f96-bf01-d314c6d77ebe-openshift-service-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:14:04.976994 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:04.976822    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fe54b2d-7431-4307-92b9-abec296278c9-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:14:04.976994 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:04.976835    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f433036-b7e8-4d47-a458-52598d130b31-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:14:04.976994 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:04.976843    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqbff\" (UniqueName: \"kubernetes.io/projected/7f433036-b7e8-4d47-a458-52598d130b31-kube-api-access-jqbff\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:14:04.976994 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:04.976852    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed67adaa-842e-4f96-bf01-d314c6d77ebe-maas-api-tls\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:14:04.978553 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:04.978531    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed67adaa-842e-4f96-bf01-d314c6d77ebe-kube-api-access-wbmsf" (OuterVolumeSpecName: "kube-api-access-wbmsf") pod "ed67adaa-842e-4f96-bf01-d314c6d77ebe" (UID: "ed67adaa-842e-4f96-bf01-d314c6d77ebe"). InnerVolumeSpecName "kube-api-access-wbmsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 02 10:14:05.078120 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:05.078055    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbmsf\" (UniqueName: \"kubernetes.io/projected/ed67adaa-842e-4f96-bf01-d314c6d77ebe-kube-api-access-wbmsf\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:14:05.496970 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:05.496937    2564 generic.go:356] "Generic (PLEG): container finished" podID="7f433036-b7e8-4d47-a458-52598d130b31" containerID="25c615967327ae8687fbac4e87c2787219509f6fc1d060af241c0caee3a05aef" exitCode=0
Aug 02 10:14:05.497152 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:05.496991    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7e5ef7e7-69fdb44446-9z5gn"
Aug 02 10:14:05.497152 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:05.497024    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7e5ef7e7-69fdb44446-9z5gn" event={"ID":"7f433036-b7e8-4d47-a458-52598d130b31","Type":"ContainerDied","Data":"25c615967327ae8687fbac4e87c2787219509f6fc1d060af241c0caee3a05aef"}
Aug 02 10:14:05.497152 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:05.497093    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7e5ef7e7-69fdb44446-9z5gn" event={"ID":"7f433036-b7e8-4d47-a458-52598d130b31","Type":"ContainerDied","Data":"01b55661d9b9cb9101ba9b830a9822b485fae80e91f424fc1cb308d5e2ea35c9"}
Aug 02 10:14:05.497152 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:05.497132    2564 scope.go:117] "RemoveContainer" containerID="25c615967327ae8687fbac4e87c2787219509f6fc1d060af241c0caee3a05aef"
Aug 02 10:14:05.498311 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:05.498285    2564 generic.go:356] "Generic (PLEG): container finished" podID="ed67adaa-842e-4f96-bf01-d314c6d77ebe" containerID="cc76ac1b3e0baf00414c21979b0e73fa633389feea9907010b4b7143f2128114" exitCode=0
Aug 02 10:14:05.498432 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:05.498322    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e5ef7e7-64b4c9f6fb-kbsxs"
Aug 02 10:14:05.498432 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:05.498351    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e5ef7e7-64b4c9f6fb-kbsxs" event={"ID":"ed67adaa-842e-4f96-bf01-d314c6d77ebe","Type":"ContainerDied","Data":"cc76ac1b3e0baf00414c21979b0e73fa633389feea9907010b4b7143f2128114"}
Aug 02 10:14:05.498432 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:05.498396    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e5ef7e7-64b4c9f6fb-kbsxs" event={"ID":"ed67adaa-842e-4f96-bf01-d314c6d77ebe","Type":"ContainerDied","Data":"cb7dfc68629d028fd61727932454194d58095fac28abe54dc1ae7402b6c6ded7"}
Aug 02 10:14:05.499924 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:05.499904    2564 generic.go:356] "Generic (PLEG): container finished" podID="3fe54b2d-7431-4307-92b9-abec296278c9" containerID="99bc734788e96a8faa3a0eb1a53e527a45092cb328b25149568adc8ee76f3d77" exitCode=0
Aug 02 10:14:05.500026 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:05.499935    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e5ef7e7-76985bcbb8-g6s8x"
Aug 02 10:14:05.500026 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:05.499959    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e5ef7e7-76985bcbb8-g6s8x" event={"ID":"3fe54b2d-7431-4307-92b9-abec296278c9","Type":"ContainerDied","Data":"99bc734788e96a8faa3a0eb1a53e527a45092cb328b25149568adc8ee76f3d77"}
Aug 02 10:14:05.500026 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:05.499999    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e5ef7e7-76985bcbb8-g6s8x" event={"ID":"3fe54b2d-7431-4307-92b9-abec296278c9","Type":"ContainerDied","Data":"d1b5c699d817edc3b1a4b1fd7ca4d7148e1ba7d2af41341f8e8ec13d1a9e0ad2"}
Aug 02 10:14:05.506015 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:05.505997    2564 scope.go:117] "RemoveContainer" containerID="25c615967327ae8687fbac4e87c2787219509f6fc1d060af241c0caee3a05aef"
Aug 02 10:14:05.506330 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:14:05.506255    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25c615967327ae8687fbac4e87c2787219509f6fc1d060af241c0caee3a05aef\": container with ID starting with 25c615967327ae8687fbac4e87c2787219509f6fc1d060af241c0caee3a05aef not found: ID does not exist" containerID="25c615967327ae8687fbac4e87c2787219509f6fc1d060af241c0caee3a05aef"
Aug 02 10:14:05.506330 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:05.506303    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25c615967327ae8687fbac4e87c2787219509f6fc1d060af241c0caee3a05aef"} err="failed to get container status \"25c615967327ae8687fbac4e87c2787219509f6fc1d060af241c0caee3a05aef\": rpc error: code = NotFound desc = could not find container \"25c615967327ae8687fbac4e87c2787219509f6fc1d060af241c0caee3a05aef\": container with ID starting with 25c615967327ae8687fbac4e87c2787219509f6fc1d060af241c0caee3a05aef not found: ID does not exist"
Aug 02 10:14:05.506330 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:05.506325    2564 scope.go:117] "RemoveContainer" containerID="cc76ac1b3e0baf00414c21979b0e73fa633389feea9907010b4b7143f2128114"
Aug 02 10:14:05.513536 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:05.513515    2564 scope.go:117] "RemoveContainer" containerID="cc76ac1b3e0baf00414c21979b0e73fa633389feea9907010b4b7143f2128114"
Aug 02 10:14:05.513933 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:14:05.513903    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc76ac1b3e0baf00414c21979b0e73fa633389feea9907010b4b7143f2128114\": container with ID starting with cc76ac1b3e0baf00414c21979b0e73fa633389feea9907010b4b7143f2128114 not found: ID does not exist" containerID="cc76ac1b3e0baf00414c21979b0e73fa633389feea9907010b4b7143f2128114"
Aug 02 10:14:05.514014 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:05.513942    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc76ac1b3e0baf00414c21979b0e73fa633389feea9907010b4b7143f2128114"} err="failed to get container status \"cc76ac1b3e0baf00414c21979b0e73fa633389feea9907010b4b7143f2128114\": rpc error: code = NotFound desc = could not find container \"cc76ac1b3e0baf00414c21979b0e73fa633389feea9907010b4b7143f2128114\": container with ID starting with cc76ac1b3e0baf00414c21979b0e73fa633389feea9907010b4b7143f2128114 not found: ID does not exist"
Aug 02 10:14:05.514014 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:05.513966    2564 scope.go:117] "RemoveContainer" containerID="99bc734788e96a8faa3a0eb1a53e527a45092cb328b25149568adc8ee76f3d77"
Aug 02 10:14:05.524156 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:05.524126    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7e5ef7e7-69fdb44446-9z5gn"]
Aug 02 10:14:05.527752 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:05.527719    2564 scope.go:117] "RemoveContainer" containerID="99bc734788e96a8faa3a0eb1a53e527a45092cb328b25149568adc8ee76f3d77"
Aug 02 10:14:05.528156 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:14:05.528131    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99bc734788e96a8faa3a0eb1a53e527a45092cb328b25149568adc8ee76f3d77\": container with ID starting with 99bc734788e96a8faa3a0eb1a53e527a45092cb328b25149568adc8ee76f3d77 not found: ID does not exist" containerID="99bc734788e96a8faa3a0eb1a53e527a45092cb328b25149568adc8ee76f3d77"
Aug 02 10:14:05.528261 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:05.528166    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99bc734788e96a8faa3a0eb1a53e527a45092cb328b25149568adc8ee76f3d77"} err="failed to get container status \"99bc734788e96a8faa3a0eb1a53e527a45092cb328b25149568adc8ee76f3d77\": rpc error: code = NotFound desc = could not find container \"99bc734788e96a8faa3a0eb1a53e527a45092cb328b25149568adc8ee76f3d77\": container with ID starting with 99bc734788e96a8faa3a0eb1a53e527a45092cb328b25149568adc8ee76f3d77 not found: ID does not exist"
Aug 02 10:14:05.529681 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:05.529664    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7e5ef7e7-69fdb44446-9z5gn"]
Aug 02 10:14:05.541739 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:05.541720    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7e5ef7e7-76985bcbb8-g6s8x"]
Aug 02 10:14:05.546289 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:05.546270    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7e5ef7e7-76985bcbb8-g6s8x"]
Aug 02 10:14:05.558221 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:05.558203    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7e5ef7e7-64b4c9f6fb-kbsxs"]
Aug 02 10:14:05.563471 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:05.563454    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7e5ef7e7-64b4c9f6fb-kbsxs"]
Aug 02 10:14:05.659108 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:05.659080    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:14:06.880384 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:06.880358    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fe54b2d-7431-4307-92b9-abec296278c9" path="/var/lib/kubelet/pods/3fe54b2d-7431-4307-92b9-abec296278c9/volumes"
Aug 02 10:14:06.880753 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:06.880738    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f433036-b7e8-4d47-a458-52598d130b31" path="/var/lib/kubelet/pods/7f433036-b7e8-4d47-a458-52598d130b31/volumes"
Aug 02 10:14:06.881123 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:06.881110    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed67adaa-842e-4f96-bf01-d314c6d77ebe" path="/var/lib/kubelet/pods/ed67adaa-842e-4f96-bf01-d314c6d77ebe/volumes"
Aug 02 10:14:07.828895 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:07.828857    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:14:10.184525 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:10.184490    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:14:47.497759 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:47.497726    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:14:48.389079 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:48.389044    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:14:54.208292 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:14:54.208197    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:15:09.342921 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:15:09.342887    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:15:09.428458 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:15:09.428426    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:15:09.639400 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:15:09.639362    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:15:09.892319 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:15:09.892210    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:15:21.343150 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:15:21.343111    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:15:21.449658 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:15:21.449624    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:15:22.243909 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:15:22.243878    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:15:22.422535 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:15:22.422510    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:15:22.582176 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:15:22.582095    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:15:23.822892 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:15:23.822856    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:15:34.229017 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:15:34.228972    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:15:34.394998 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:15:34.394960    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:15:35.064583 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:15:35.064546    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:15:39.557222 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:15:39.557184    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:15:39.664881 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:15:39.664848    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:15:43.589361 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:15:43.589322    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:15:43.770128 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:15:43.770091    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:15:44.010893 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:15:44.010856    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:15:54.419590 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:15:54.419547    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:15:54.652592 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:15:54.652528    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:16:15.650039 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:15.650007    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:16:22.635649 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:22.635556    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:16:22.738908 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:22.738877    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:16:23.015605 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:23.015517    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:16:23.388640 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:23.388611    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:16:23.719389 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:23.719312    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:16:26.579870 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:26.579834    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:16:26.684122 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:26.684087    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:16:27.748948 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:27.748915    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e3a61d-5b869b769-vfwx7"]
Aug 02 10:16:27.749321 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:27.749218    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f433036-b7e8-4d47-a458-52598d130b31" containerName="payload-processing"
Aug 02 10:16:27.749321 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:27.749242    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f433036-b7e8-4d47-a458-52598d130b31" containerName="payload-processing"
Aug 02 10:16:27.749321 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:27.749254    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed67adaa-842e-4f96-bf01-d314c6d77ebe" containerName="maas-api"
Aug 02 10:16:27.749321 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:27.749260    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed67adaa-842e-4f96-bf01-d314c6d77ebe" containerName="maas-api"
Aug 02 10:16:27.749321 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:27.749270    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fe54b2d-7431-4307-92b9-abec296278c9" containerName="payload-pre-processing"
Aug 02 10:16:27.749321 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:27.749275    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fe54b2d-7431-4307-92b9-abec296278c9" containerName="payload-pre-processing"
Aug 02 10:16:27.749523 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:27.749330    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fe54b2d-7431-4307-92b9-abec296278c9" containerName="payload-pre-processing"
Aug 02 10:16:27.749523 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:27.749337    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed67adaa-842e-4f96-bf01-d314c6d77ebe" containerName="maas-api"
Aug 02 10:16:27.749523 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:27.749343    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f433036-b7e8-4d47-a458-52598d130b31" containerName="payload-processing"
Aug 02 10:16:27.752106 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:27.752089    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3a61d-5b869b769-vfwx7"
Aug 02 10:16:27.754287 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:27.754260    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-e3a61d-serving-cert\""
Aug 02 10:16:27.757585 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:27.757560    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcvvw\" (UniqueName: \"kubernetes.io/projected/21064ba7-7cb3-4cbb-8da0-635060293e70-kube-api-access-wcvvw\") pod \"maas-api-e2e-inf-a-e3a61d-5b869b769-vfwx7\" (UID: \"21064ba7-7cb3-4cbb-8da0-635060293e70\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3a61d-5b869b769-vfwx7"
Aug 02 10:16:27.757700 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:27.757643    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21064ba7-7cb3-4cbb-8da0-635060293e70-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e3a61d-5b869b769-vfwx7\" (UID: \"21064ba7-7cb3-4cbb-8da0-635060293e70\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3a61d-5b869b769-vfwx7"
Aug 02 10:16:27.757700 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:27.757681    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21064ba7-7cb3-4cbb-8da0-635060293e70-maas-api-tls\") pod \"maas-api-e2e-inf-a-e3a61d-5b869b769-vfwx7\" (UID: \"21064ba7-7cb3-4cbb-8da0-635060293e70\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3a61d-5b869b769-vfwx7"
Aug 02 10:16:27.775438 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:27.775415    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e3a61d-5b869b769-vfwx7"]
Aug 02 10:16:27.797029 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:27.796992    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:16:27.858023 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:27.857997    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21064ba7-7cb3-4cbb-8da0-635060293e70-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e3a61d-5b869b769-vfwx7\" (UID: \"21064ba7-7cb3-4cbb-8da0-635060293e70\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3a61d-5b869b769-vfwx7"
Aug 02 10:16:27.858125 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:27.858030    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21064ba7-7cb3-4cbb-8da0-635060293e70-maas-api-tls\") pod \"maas-api-e2e-inf-a-e3a61d-5b869b769-vfwx7\" (UID: \"21064ba7-7cb3-4cbb-8da0-635060293e70\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3a61d-5b869b769-vfwx7"
Aug 02 10:16:27.858125 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:27.858079    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcvvw\" (UniqueName: \"kubernetes.io/projected/21064ba7-7cb3-4cbb-8da0-635060293e70-kube-api-access-wcvvw\") pod \"maas-api-e2e-inf-a-e3a61d-5b869b769-vfwx7\" (UID: \"21064ba7-7cb3-4cbb-8da0-635060293e70\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3a61d-5b869b769-vfwx7"
Aug 02 10:16:27.858224 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:16:27.858202    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-e3a61d-serving-cert: secret "maas-api-e2e-inf-a-e3a61d-serving-cert" not found
Aug 02 10:16:27.858320 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:16:27.858309    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21064ba7-7cb3-4cbb-8da0-635060293e70-maas-api-tls podName:21064ba7-7cb3-4cbb-8da0-635060293e70 nodeName:}" failed. No retries permitted until 2026-08-02 10:16:28.358290081 +0000 UTC m=+2612.073095524 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/21064ba7-7cb3-4cbb-8da0-635060293e70-maas-api-tls") pod "maas-api-e2e-inf-a-e3a61d-5b869b769-vfwx7" (UID: "21064ba7-7cb3-4cbb-8da0-635060293e70") : secret "maas-api-e2e-inf-a-e3a61d-serving-cert" not found
Aug 02 10:16:27.858677 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:27.858661    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21064ba7-7cb3-4cbb-8da0-635060293e70-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e3a61d-5b869b769-vfwx7\" (UID: \"21064ba7-7cb3-4cbb-8da0-635060293e70\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3a61d-5b869b769-vfwx7"
Aug 02 10:16:27.883925 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:27.883899    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcvvw\" (UniqueName: \"kubernetes.io/projected/21064ba7-7cb3-4cbb-8da0-635060293e70-kube-api-access-wcvvw\") pod \"maas-api-e2e-inf-a-e3a61d-5b869b769-vfwx7\" (UID: \"21064ba7-7cb3-4cbb-8da0-635060293e70\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3a61d-5b869b769-vfwx7"
Aug 02 10:16:28.200626 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:28.200586    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:16:28.361650 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:28.361615    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21064ba7-7cb3-4cbb-8da0-635060293e70-maas-api-tls\") pod \"maas-api-e2e-inf-a-e3a61d-5b869b769-vfwx7\" (UID: \"21064ba7-7cb3-4cbb-8da0-635060293e70\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3a61d-5b869b769-vfwx7"
Aug 02 10:16:28.363932 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:28.363907    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21064ba7-7cb3-4cbb-8da0-635060293e70-maas-api-tls\") pod \"maas-api-e2e-inf-a-e3a61d-5b869b769-vfwx7\" (UID: \"21064ba7-7cb3-4cbb-8da0-635060293e70\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3a61d-5b869b769-vfwx7"
Aug 02 10:16:28.383938 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:28.383910    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:16:28.481354 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:28.481275    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:16:28.513895 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:28.513870    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e3a61d-68c7cd9cb6-46bbs"]
Aug 02 10:16:28.517205 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:28.517191    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e3a61d-68c7cd9cb6-46bbs"
Aug 02 10:16:28.519353 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:28.519333    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-e3a61d\""
Aug 02 10:16:28.519784 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:28.519751    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-e3a61d-dockercfg-l7mr8\""
Aug 02 10:16:28.535760 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:28.535740    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e3a61d-68c7cd9cb6-46bbs"]
Aug 02 10:16:28.563166 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:28.563146    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds6v6\" (UniqueName: \"kubernetes.io/projected/41523f51-825d-4ac7-abf5-1a38afd5b01c-kube-api-access-ds6v6\") pod \"payload-processing-e2e-inf-a-e3a61d-68c7cd9cb6-46bbs\" (UID: \"41523f51-825d-4ac7-abf5-1a38afd5b01c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e3a61d-68c7cd9cb6-46bbs"
Aug 02 10:16:28.563296 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:28.563187    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41523f51-825d-4ac7-abf5-1a38afd5b01c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e3a61d-68c7cd9cb6-46bbs\" (UID: \"41523f51-825d-4ac7-abf5-1a38afd5b01c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e3a61d-68c7cd9cb6-46bbs"
Aug 02 10:16:28.603566 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:28.603544    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e3a61d-6677bdb548-qxhpt"]
Aug 02 10:16:28.606674 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:28.606660    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3a61d-6677bdb548-qxhpt"
Aug 02 10:16:28.617629 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:28.617607    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e3a61d-6677bdb548-qxhpt"]
Aug 02 10:16:28.661938 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:28.661911    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3a61d-5b869b769-vfwx7"
Aug 02 10:16:28.664308 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:28.664283    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds6v6\" (UniqueName: \"kubernetes.io/projected/41523f51-825d-4ac7-abf5-1a38afd5b01c-kube-api-access-ds6v6\") pod \"payload-processing-e2e-inf-a-e3a61d-68c7cd9cb6-46bbs\" (UID: \"41523f51-825d-4ac7-abf5-1a38afd5b01c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e3a61d-68c7cd9cb6-46bbs"
Aug 02 10:16:28.664417 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:28.664337    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q46bq\" (UniqueName: \"kubernetes.io/projected/a67f2645-4322-431f-903f-02cdc067e636-kube-api-access-q46bq\") pod \"payload-pre-processing-e2e-inf-a-e3a61d-6677bdb548-qxhpt\" (UID: \"a67f2645-4322-431f-903f-02cdc067e636\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3a61d-6677bdb548-qxhpt"
Aug 02 10:16:28.664417 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:28.664377    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a67f2645-4322-431f-903f-02cdc067e636-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e3a61d-6677bdb548-qxhpt\" (UID: \"a67f2645-4322-431f-903f-02cdc067e636\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3a61d-6677bdb548-qxhpt"
Aug 02 10:16:28.664512 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:28.664422    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41523f51-825d-4ac7-abf5-1a38afd5b01c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e3a61d-68c7cd9cb6-46bbs\" (UID: \"41523f51-825d-4ac7-abf5-1a38afd5b01c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e3a61d-68c7cd9cb6-46bbs"
Aug 02 10:16:28.665421 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:28.665207    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41523f51-825d-4ac7-abf5-1a38afd5b01c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e3a61d-68c7cd9cb6-46bbs\" (UID: \"41523f51-825d-4ac7-abf5-1a38afd5b01c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e3a61d-68c7cd9cb6-46bbs"
Aug 02 10:16:28.676793 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:28.676755    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds6v6\" (UniqueName: \"kubernetes.io/projected/41523f51-825d-4ac7-abf5-1a38afd5b01c-kube-api-access-ds6v6\") pod \"payload-processing-e2e-inf-a-e3a61d-68c7cd9cb6-46bbs\" (UID: \"41523f51-825d-4ac7-abf5-1a38afd5b01c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e3a61d-68c7cd9cb6-46bbs"
Aug 02 10:16:28.764904 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:28.764872    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q46bq\" (UniqueName: \"kubernetes.io/projected/a67f2645-4322-431f-903f-02cdc067e636-kube-api-access-q46bq\") pod \"payload-pre-processing-e2e-inf-a-e3a61d-6677bdb548-qxhpt\" (UID: \"a67f2645-4322-431f-903f-02cdc067e636\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3a61d-6677bdb548-qxhpt"
Aug 02 10:16:28.764904 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:28.764909    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a67f2645-4322-431f-903f-02cdc067e636-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e3a61d-6677bdb548-qxhpt\" (UID: \"a67f2645-4322-431f-903f-02cdc067e636\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3a61d-6677bdb548-qxhpt"
Aug 02 10:16:28.765814 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:28.765786    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a67f2645-4322-431f-903f-02cdc067e636-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e3a61d-6677bdb548-qxhpt\" (UID: \"a67f2645-4322-431f-903f-02cdc067e636\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3a61d-6677bdb548-qxhpt"
Aug 02 10:16:28.774082 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:28.774058    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q46bq\" (UniqueName: \"kubernetes.io/projected/a67f2645-4322-431f-903f-02cdc067e636-kube-api-access-q46bq\") pod \"payload-pre-processing-e2e-inf-a-e3a61d-6677bdb548-qxhpt\" (UID: \"a67f2645-4322-431f-903f-02cdc067e636\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3a61d-6677bdb548-qxhpt"
Aug 02 10:16:28.787858 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:28.787836    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e3a61d-5b869b769-vfwx7"]
Aug 02 10:16:28.789849 ip-10-0-141-232 kubenswrapper[2564]: W0802 10:16:28.789827    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21064ba7_7cb3_4cbb_8da0_635060293e70.slice/crio-530f7aa081492cf225a73577cf85f91ce1b4d6ede42509e7f298377110c6d4b4 WatchSource:0}: Error finding container 530f7aa081492cf225a73577cf85f91ce1b4d6ede42509e7f298377110c6d4b4: Status 404 returned error can't find the container with id 530f7aa081492cf225a73577cf85f91ce1b4d6ede42509e7f298377110c6d4b4
Aug 02 10:16:28.791266 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:28.791226    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 02 10:16:28.828218 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:28.828199    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e3a61d-68c7cd9cb6-46bbs"
Aug 02 10:16:28.916538 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:28.916511    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3a61d-6677bdb548-qxhpt"
Aug 02 10:16:28.964838 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:28.964807    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e3a61d-68c7cd9cb6-46bbs"]
Aug 02 10:16:28.970290 ip-10-0-141-232 kubenswrapper[2564]: W0802 10:16:28.970096    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41523f51_825d_4ac7_abf5_1a38afd5b01c.slice/crio-fdcb35439f9ce2f6232e79ff4d4d8f2bc302804abf527ef1df53358419ed18cb WatchSource:0}: Error finding container fdcb35439f9ce2f6232e79ff4d4d8f2bc302804abf527ef1df53358419ed18cb: Status 404 returned error can't find the container with id fdcb35439f9ce2f6232e79ff4d4d8f2bc302804abf527ef1df53358419ed18cb
Aug 02 10:16:29.059666 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:29.059635    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e3a61d-68c7cd9cb6-46bbs" event={"ID":"41523f51-825d-4ac7-abf5-1a38afd5b01c","Type":"ContainerStarted","Data":"a445415933debbedcedb3490f69bfd8c8b73e0e83db92aa90bbf852fd592c620"}
Aug 02 10:16:29.059817 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:29.059691    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e3a61d-68c7cd9cb6-46bbs" event={"ID":"41523f51-825d-4ac7-abf5-1a38afd5b01c","Type":"ContainerStarted","Data":"fdcb35439f9ce2f6232e79ff4d4d8f2bc302804abf527ef1df53358419ed18cb"}
Aug 02 10:16:29.059817 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:29.059722    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e3a61d-68c7cd9cb6-46bbs"
Aug 02 10:16:29.060945 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:29.060912    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3a61d-5b869b769-vfwx7" event={"ID":"21064ba7-7cb3-4cbb-8da0-635060293e70","Type":"ContainerStarted","Data":"530f7aa081492cf225a73577cf85f91ce1b4d6ede42509e7f298377110c6d4b4"}
Aug 02 10:16:29.064549 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:29.064526    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e3a61d-6677bdb548-qxhpt"]
Aug 02 10:16:29.067376 ip-10-0-141-232 kubenswrapper[2564]: W0802 10:16:29.067345    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda67f2645_4322_431f_903f_02cdc067e636.slice/crio-669f30599f2f7036f145b3d04a8893f2670a42cfb75423a782c1affc21c490e7 WatchSource:0}: Error finding container 669f30599f2f7036f145b3d04a8893f2670a42cfb75423a782c1affc21c490e7: Status 404 returned error can't find the container with id 669f30599f2f7036f145b3d04a8893f2670a42cfb75423a782c1affc21c490e7
Aug 02 10:16:29.081774 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:29.081737    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-e3a61d-68c7cd9cb6-46bbs" podStartSLOduration=1.081724419 podStartE2EDuration="1.081724419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:16:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:16:29.078517923 +0000 UTC m=+2612.793323364" watchObservedRunningTime="2026-08-02 10:16:29.081724419 +0000 UTC m=+2612.796529845"
Aug 02 10:16:30.068018 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:30.067982    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3a61d-5b869b769-vfwx7" event={"ID":"21064ba7-7cb3-4cbb-8da0-635060293e70","Type":"ContainerStarted","Data":"b1fb5568bb69ee996159a81494da3f9de9555c7ed26abe6e963bb0cad20b6b27"}
Aug 02 10:16:30.068462 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:30.068080    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3a61d-5b869b769-vfwx7"
Aug 02 10:16:30.069580 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:30.069551    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3a61d-6677bdb548-qxhpt" event={"ID":"a67f2645-4322-431f-903f-02cdc067e636","Type":"ContainerStarted","Data":"400750cd8e3a740e66008443258abdc1dbbcd8ec36fbddb0dc6265e4ceab0975"}
Aug 02 10:16:30.069730 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:30.069601    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3a61d-6677bdb548-qxhpt" event={"ID":"a67f2645-4322-431f-903f-02cdc067e636","Type":"ContainerStarted","Data":"669f30599f2f7036f145b3d04a8893f2670a42cfb75423a782c1affc21c490e7"}
Aug 02 10:16:30.069730 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:30.069697    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3a61d-6677bdb548-qxhpt"
Aug 02 10:16:30.100646 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:30.100594    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3a61d-5b869b769-vfwx7" podStartSLOduration=2.624755976 podStartE2EDuration="3.100579166s" totalImagesPullingTime="475.82319ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:16:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:16:30.100414264 +0000 UTC m=+2613.815219719" watchObservedRunningTime="2026-08-02 10:16:30.100579166 +0000 UTC m=+2613.815384617"
Aug 02 10:16:30.121586 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:30.121544    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3a61d-6677bdb548-qxhpt" podStartSLOduration=2.121530579 podStartE2EDuration="2.121530579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:16:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:16:30.120741883 +0000 UTC m=+2613.835547331" watchObservedRunningTime="2026-08-02 10:16:30.121530579 +0000 UTC m=+2613.836336230"
Aug 02 10:16:36.078744 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:36.078710    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3a61d-5b869b769-vfwx7"
Aug 02 10:16:40.071880 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:40.071848    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e3a61d-68c7cd9cb6-46bbs"
Aug 02 10:16:41.075556 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:41.075528    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3a61d-6677bdb548-qxhpt"
Aug 02 10:16:51.496414 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:51.496374    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:16:51.825453 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:51.825358    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:16:52.175526 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:52.175497    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:16:52.855467 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:52.855437    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:16:55.530809 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:55.530774    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:16:55.620878 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:55.620836    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:16:56.703935 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:56.703898    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6c8adc-845494bb57-vb7q8"]
Aug 02 10:16:56.707262 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:56.707226    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6c8adc-845494bb57-vb7q8"
Aug 02 10:16:56.712390 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:56.712371    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-6c8adc-serving-cert\""
Aug 02 10:16:56.719560 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:56.719537    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6c8adc-845494bb57-vb7q8"]
Aug 02 10:16:56.748634 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:56.748596    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:16:56.782413 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:56.782380    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/691f2e6f-fecf-4910-8a08-ed6eecb4e48e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6c8adc-845494bb57-vb7q8\" (UID: \"691f2e6f-fecf-4910-8a08-ed6eecb4e48e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6c8adc-845494bb57-vb7q8"
Aug 02 10:16:56.782556 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:56.782419    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkbmh\" (UniqueName: \"kubernetes.io/projected/691f2e6f-fecf-4910-8a08-ed6eecb4e48e-kube-api-access-hkbmh\") pod \"maas-api-e2e-inf-b-6c8adc-845494bb57-vb7q8\" (UID: \"691f2e6f-fecf-4910-8a08-ed6eecb4e48e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6c8adc-845494bb57-vb7q8"
Aug 02 10:16:56.782556 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:56.782465    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/691f2e6f-fecf-4910-8a08-ed6eecb4e48e-maas-api-tls\") pod \"maas-api-e2e-inf-b-6c8adc-845494bb57-vb7q8\" (UID: \"691f2e6f-fecf-4910-8a08-ed6eecb4e48e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6c8adc-845494bb57-vb7q8"
Aug 02 10:16:56.883834 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:56.883796    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/691f2e6f-fecf-4910-8a08-ed6eecb4e48e-maas-api-tls\") pod \"maas-api-e2e-inf-b-6c8adc-845494bb57-vb7q8\" (UID: \"691f2e6f-fecf-4910-8a08-ed6eecb4e48e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6c8adc-845494bb57-vb7q8"
Aug 02 10:16:56.883991 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:56.883927    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/691f2e6f-fecf-4910-8a08-ed6eecb4e48e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6c8adc-845494bb57-vb7q8\" (UID: \"691f2e6f-fecf-4910-8a08-ed6eecb4e48e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6c8adc-845494bb57-vb7q8"
Aug 02 10:16:56.883991 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:56.883959    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkbmh\" (UniqueName: \"kubernetes.io/projected/691f2e6f-fecf-4910-8a08-ed6eecb4e48e-kube-api-access-hkbmh\") pod \"maas-api-e2e-inf-b-6c8adc-845494bb57-vb7q8\" (UID: \"691f2e6f-fecf-4910-8a08-ed6eecb4e48e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6c8adc-845494bb57-vb7q8"
Aug 02 10:16:56.884447 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:16:56.884425    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-6c8adc-serving-cert: secret "maas-api-e2e-inf-b-6c8adc-serving-cert" not found
Aug 02 10:16:56.884551 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:16:56.884503    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/691f2e6f-fecf-4910-8a08-ed6eecb4e48e-maas-api-tls podName:691f2e6f-fecf-4910-8a08-ed6eecb4e48e nodeName:}" failed. No retries permitted until 2026-08-02 10:16:57.384481017 +0000 UTC m=+2641.099286443 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/691f2e6f-fecf-4910-8a08-ed6eecb4e48e-maas-api-tls") pod "maas-api-e2e-inf-b-6c8adc-845494bb57-vb7q8" (UID: "691f2e6f-fecf-4910-8a08-ed6eecb4e48e") : secret "maas-api-e2e-inf-b-6c8adc-serving-cert" not found
Aug 02 10:16:56.885389 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:56.885361    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/691f2e6f-fecf-4910-8a08-ed6eecb4e48e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6c8adc-845494bb57-vb7q8\" (UID: \"691f2e6f-fecf-4910-8a08-ed6eecb4e48e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6c8adc-845494bb57-vb7q8"
Aug 02 10:16:56.894912 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:56.894882    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkbmh\" (UniqueName: \"kubernetes.io/projected/691f2e6f-fecf-4910-8a08-ed6eecb4e48e-kube-api-access-hkbmh\") pod \"maas-api-e2e-inf-b-6c8adc-845494bb57-vb7q8\" (UID: \"691f2e6f-fecf-4910-8a08-ed6eecb4e48e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6c8adc-845494bb57-vb7q8"
Aug 02 10:16:57.094903 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:57.094825    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:16:57.301991 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:57.301967    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:16:57.387106 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:57.387077    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/691f2e6f-fecf-4910-8a08-ed6eecb4e48e-maas-api-tls\") pod \"maas-api-e2e-inf-b-6c8adc-845494bb57-vb7q8\" (UID: \"691f2e6f-fecf-4910-8a08-ed6eecb4e48e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6c8adc-845494bb57-vb7q8"
Aug 02 10:16:57.389331 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:57.389307    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/691f2e6f-fecf-4910-8a08-ed6eecb4e48e-maas-api-tls\") pod \"maas-api-e2e-inf-b-6c8adc-845494bb57-vb7q8\" (UID: \"691f2e6f-fecf-4910-8a08-ed6eecb4e48e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6c8adc-845494bb57-vb7q8"
Aug 02 10:16:57.401960 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:57.401936    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:16:57.438496 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:57.438468    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6c8adc-5d47ddc54b-44wsx"]
Aug 02 10:16:57.441750 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:57.441732    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6c8adc-5d47ddc54b-44wsx"
Aug 02 10:16:57.444777 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:57.444754    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-6c8adc-dockercfg-45pds\""
Aug 02 10:16:57.444868 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:57.444797    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-6c8adc\""
Aug 02 10:16:57.456727 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:57.456707    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6c8adc-5d47ddc54b-44wsx"]
Aug 02 10:16:57.488374 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:57.488345    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cab6d9f3-ef7b-4b72-a578-f05013dc2f37-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6c8adc-5d47ddc54b-44wsx\" (UID: \"cab6d9f3-ef7b-4b72-a578-f05013dc2f37\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6c8adc-5d47ddc54b-44wsx"
Aug 02 10:16:57.488478 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:57.488407    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w65kg\" (UniqueName: \"kubernetes.io/projected/cab6d9f3-ef7b-4b72-a578-f05013dc2f37-kube-api-access-w65kg\") pod \"payload-processing-e2e-inf-b-6c8adc-5d47ddc54b-44wsx\" (UID: \"cab6d9f3-ef7b-4b72-a578-f05013dc2f37\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6c8adc-5d47ddc54b-44wsx"
Aug 02 10:16:57.493900 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:57.493879    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6c8adc-6f7995f884-j4ph8"]
Aug 02 10:16:57.497019 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:57.497003    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6c8adc-6f7995f884-j4ph8"
Aug 02 10:16:57.513533 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:57.513511    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6c8adc-6f7995f884-j4ph8"]
Aug 02 10:16:57.589290 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:57.589252    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w65kg\" (UniqueName: \"kubernetes.io/projected/cab6d9f3-ef7b-4b72-a578-f05013dc2f37-kube-api-access-w65kg\") pod \"payload-processing-e2e-inf-b-6c8adc-5d47ddc54b-44wsx\" (UID: \"cab6d9f3-ef7b-4b72-a578-f05013dc2f37\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6c8adc-5d47ddc54b-44wsx"
Aug 02 10:16:57.589469 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:57.589304    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn55c\" (UniqueName: \"kubernetes.io/projected/d686e6ac-b668-42ca-bf58-7659ba892f11-kube-api-access-qn55c\") pod \"payload-pre-processing-e2e-inf-b-6c8adc-6f7995f884-j4ph8\" (UID: \"d686e6ac-b668-42ca-bf58-7659ba892f11\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6c8adc-6f7995f884-j4ph8"
Aug 02 10:16:57.589469 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:57.589330    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d686e6ac-b668-42ca-bf58-7659ba892f11-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6c8adc-6f7995f884-j4ph8\" (UID: \"d686e6ac-b668-42ca-bf58-7659ba892f11\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6c8adc-6f7995f884-j4ph8"
Aug 02 10:16:57.589469 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:57.589365    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cab6d9f3-ef7b-4b72-a578-f05013dc2f37-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6c8adc-5d47ddc54b-44wsx\" (UID: \"cab6d9f3-ef7b-4b72-a578-f05013dc2f37\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6c8adc-5d47ddc54b-44wsx"
Aug 02 10:16:57.589966 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:57.589947    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cab6d9f3-ef7b-4b72-a578-f05013dc2f37-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6c8adc-5d47ddc54b-44wsx\" (UID: \"cab6d9f3-ef7b-4b72-a578-f05013dc2f37\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6c8adc-5d47ddc54b-44wsx"
Aug 02 10:16:57.599144 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:57.599120    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w65kg\" (UniqueName: \"kubernetes.io/projected/cab6d9f3-ef7b-4b72-a578-f05013dc2f37-kube-api-access-w65kg\") pod \"payload-processing-e2e-inf-b-6c8adc-5d47ddc54b-44wsx\" (UID: \"cab6d9f3-ef7b-4b72-a578-f05013dc2f37\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6c8adc-5d47ddc54b-44wsx"
Aug 02 10:16:57.618678 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:57.618649    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6c8adc-845494bb57-vb7q8"
Aug 02 10:16:57.690212 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:57.690180    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn55c\" (UniqueName: \"kubernetes.io/projected/d686e6ac-b668-42ca-bf58-7659ba892f11-kube-api-access-qn55c\") pod \"payload-pre-processing-e2e-inf-b-6c8adc-6f7995f884-j4ph8\" (UID: \"d686e6ac-b668-42ca-bf58-7659ba892f11\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6c8adc-6f7995f884-j4ph8"
Aug 02 10:16:57.690533 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:57.690515    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d686e6ac-b668-42ca-bf58-7659ba892f11-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6c8adc-6f7995f884-j4ph8\" (UID: \"d686e6ac-b668-42ca-bf58-7659ba892f11\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6c8adc-6f7995f884-j4ph8"
Aug 02 10:16:57.691159 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:57.691135    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d686e6ac-b668-42ca-bf58-7659ba892f11-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6c8adc-6f7995f884-j4ph8\" (UID: \"d686e6ac-b668-42ca-bf58-7659ba892f11\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6c8adc-6f7995f884-j4ph8"
Aug 02 10:16:57.699083 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:57.699062    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn55c\" (UniqueName: \"kubernetes.io/projected/d686e6ac-b668-42ca-bf58-7659ba892f11-kube-api-access-qn55c\") pod \"payload-pre-processing-e2e-inf-b-6c8adc-6f7995f884-j4ph8\" (UID: \"d686e6ac-b668-42ca-bf58-7659ba892f11\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6c8adc-6f7995f884-j4ph8"
Aug 02 10:16:57.747967 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:57.747776    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6c8adc-845494bb57-vb7q8"]
Aug 02 10:16:57.750154 ip-10-0-141-232 kubenswrapper[2564]: W0802 10:16:57.750120    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod691f2e6f_fecf_4910_8a08_ed6eecb4e48e.slice/crio-a98bde379c34d0727ec832725bdd4cd19a490f721bd02212b0d9995356252445 WatchSource:0}: Error finding container a98bde379c34d0727ec832725bdd4cd19a490f721bd02212b0d9995356252445: Status 404 returned error can't find the container with id a98bde379c34d0727ec832725bdd4cd19a490f721bd02212b0d9995356252445
Aug 02 10:16:57.751691 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:57.751671    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6c8adc-5d47ddc54b-44wsx"
Aug 02 10:16:57.806226 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:57.805920    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6c8adc-6f7995f884-j4ph8"
Aug 02 10:16:57.939459 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:57.939384    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6c8adc-6f7995f884-j4ph8"]
Aug 02 10:16:57.942456 ip-10-0-141-232 kubenswrapper[2564]: W0802 10:16:57.942426    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd686e6ac_b668_42ca_bf58_7659ba892f11.slice/crio-a5abbbf96f2b018c9ce3269ed97cb4a4fdc948c8b8d34a972c0fc0c236e04ad1 WatchSource:0}: Error finding container a5abbbf96f2b018c9ce3269ed97cb4a4fdc948c8b8d34a972c0fc0c236e04ad1: Status 404 returned error can't find the container with id a5abbbf96f2b018c9ce3269ed97cb4a4fdc948c8b8d34a972c0fc0c236e04ad1
Aug 02 10:16:58.091120 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:58.091097    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6c8adc-5d47ddc54b-44wsx"]
Aug 02 10:16:58.093326 ip-10-0-141-232 kubenswrapper[2564]: W0802 10:16:58.093290    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcab6d9f3_ef7b_4b72_a578_f05013dc2f37.slice/crio-be0c6838c7c1d03ba2c2635025b00ccc19e00c18b7fc577301fce6a06594dda3 WatchSource:0}: Error finding container be0c6838c7c1d03ba2c2635025b00ccc19e00c18b7fc577301fce6a06594dda3: Status 404 returned error can't find the container with id be0c6838c7c1d03ba2c2635025b00ccc19e00c18b7fc577301fce6a06594dda3
Aug 02 10:16:58.182142 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:58.182118    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6c8adc-6f7995f884-j4ph8" event={"ID":"d686e6ac-b668-42ca-bf58-7659ba892f11","Type":"ContainerStarted","Data":"606392366c3ac2cbb2d0bce2ffb253d6624e9585f7c391b499bc6cdd862f687e"}
Aug 02 10:16:58.182295 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:58.182171    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6c8adc-6f7995f884-j4ph8" event={"ID":"d686e6ac-b668-42ca-bf58-7659ba892f11","Type":"ContainerStarted","Data":"a5abbbf96f2b018c9ce3269ed97cb4a4fdc948c8b8d34a972c0fc0c236e04ad1"}
Aug 02 10:16:58.182295 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:58.182204    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6c8adc-6f7995f884-j4ph8"
Aug 02 10:16:58.183728 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:58.183708    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6c8adc-5d47ddc54b-44wsx" event={"ID":"cab6d9f3-ef7b-4b72-a578-f05013dc2f37","Type":"ContainerStarted","Data":"7be6d3cb76309b33bbb024947df73d1dd0620e786359925b7af1bf5c023c528d"}
Aug 02 10:16:58.183835 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:58.183744    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6c8adc-5d47ddc54b-44wsx" event={"ID":"cab6d9f3-ef7b-4b72-a578-f05013dc2f37","Type":"ContainerStarted","Data":"be0c6838c7c1d03ba2c2635025b00ccc19e00c18b7fc577301fce6a06594dda3"}
Aug 02 10:16:58.183835 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:58.183770    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-6c8adc-5d47ddc54b-44wsx"
Aug 02 10:16:58.184913 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:58.184889    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6c8adc-845494bb57-vb7q8" event={"ID":"691f2e6f-fecf-4910-8a08-ed6eecb4e48e","Type":"ContainerStarted","Data":"a98bde379c34d0727ec832725bdd4cd19a490f721bd02212b0d9995356252445"}
Aug 02 10:16:58.209704 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:58.209663    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6c8adc-6f7995f884-j4ph8" podStartSLOduration=1.209650074 podStartE2EDuration="1.209650074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:16:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:16:58.207655839 +0000 UTC m=+2641.922461286" watchObservedRunningTime="2026-08-02 10:16:58.209650074 +0000 UTC m=+2641.924455573"
Aug 02 10:16:58.237618 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:58.237580    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-6c8adc-5d47ddc54b-44wsx" podStartSLOduration=1.237565474 podStartE2EDuration="1.237565474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:16:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:16:58.23709579 +0000 UTC m=+2641.951901238" watchObservedRunningTime="2026-08-02 10:16:58.237565474 +0000 UTC m=+2641.952370923"
Aug 02 10:16:59.191021 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:59.190983    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6c8adc-845494bb57-vb7q8" event={"ID":"691f2e6f-fecf-4910-8a08-ed6eecb4e48e","Type":"ContainerStarted","Data":"4bd274a3e1ddde0edaf19b0653195d41970ac9ca8e892cee4130597924552f1d"}
Aug 02 10:16:59.191469 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:59.191060    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6c8adc-845494bb57-vb7q8"
Aug 02 10:16:59.210018 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:16:59.209975    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6c8adc-845494bb57-vb7q8" podStartSLOduration=2.766783761 podStartE2EDuration="3.209962055s" totalImagesPullingTime="443.178294ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:16:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:16:59.207878192 +0000 UTC m=+2642.922683640" watchObservedRunningTime="2026-08-02 10:16:59.209962055 +0000 UTC m=+2642.924767503"
Aug 02 10:17:05.206194 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:17:05.206167    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6c8adc-845494bb57-vb7q8"
Aug 02 10:17:09.193147 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:17:09.193112    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-6c8adc-5d47ddc54b-44wsx"
Aug 02 10:17:09.193559 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:17:09.193188    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6c8adc-6f7995f884-j4ph8"
Aug 02 10:17:20.541969 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:17:20.541931    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:17:29.189663 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:17:29.189622    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:17:47.274374 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:17:47.274341    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:18:07.089436 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:18:07.089404    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:18:55.768058 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:18:55.768025    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:18:56.840107 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:18:56.840069    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:19:01.619276 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:01.619225    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e3a61d-5b869b769-vfwx7"]
Aug 02 10:19:01.619843 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:01.619505    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3a61d-5b869b769-vfwx7" podUID="21064ba7-7cb3-4cbb-8da0-635060293e70" containerName="maas-api" containerID="cri-o://b1fb5568bb69ee996159a81494da3f9de9555c7ed26abe6e963bb0cad20b6b27" gracePeriod=30
Aug 02 10:19:01.657448 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:01.657413    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:19:01.691398 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:01.691367    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e3a61d-68c7cd9cb6-46bbs"]
Aug 02 10:19:01.691603 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:01.691581    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-e3a61d-68c7cd9cb6-46bbs" podUID="41523f51-825d-4ac7-abf5-1a38afd5b01c" containerName="payload-processing" containerID="cri-o://a445415933debbedcedb3490f69bfd8c8b73e0e83db92aa90bbf852fd592c620" gracePeriod=30
Aug 02 10:19:01.705987 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:01.705961    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e3a61d-6677bdb548-qxhpt"]
Aug 02 10:19:01.706199 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:01.706176    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3a61d-6677bdb548-qxhpt" podUID="a67f2645-4322-431f-903f-02cdc067e636" containerName="payload-pre-processing" containerID="cri-o://400750cd8e3a740e66008443258abdc1dbbcd8ec36fbddb0dc6265e4ceab0975" gracePeriod=30
Aug 02 10:19:01.765653 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:01.765621    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:19:01.871769 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:01.871606    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:19:01.940832 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:01.940584    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3a61d-5b869b769-vfwx7"
Aug 02 10:19:01.969111 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:01.969088    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e3a61d-68c7cd9cb6-46bbs"
Aug 02 10:19:01.985331 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:01.985301    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:19:02.007934 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.007891    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3a61d-6677bdb548-qxhpt"
Aug 02 10:19:02.055458 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.055437    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q46bq\" (UniqueName: \"kubernetes.io/projected/a67f2645-4322-431f-903f-02cdc067e636-kube-api-access-q46bq\") pod \"a67f2645-4322-431f-903f-02cdc067e636\" (UID: \"a67f2645-4322-431f-903f-02cdc067e636\") "
Aug 02 10:19:02.055584 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.055466    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a67f2645-4322-431f-903f-02cdc067e636-plugins-config-volume\") pod \"a67f2645-4322-431f-903f-02cdc067e636\" (UID: \"a67f2645-4322-431f-903f-02cdc067e636\") "
Aug 02 10:19:02.055584 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.055566    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ds6v6\" (UniqueName: \"kubernetes.io/projected/41523f51-825d-4ac7-abf5-1a38afd5b01c-kube-api-access-ds6v6\") pod \"41523f51-825d-4ac7-abf5-1a38afd5b01c\" (UID: \"41523f51-825d-4ac7-abf5-1a38afd5b01c\") "
Aug 02 10:19:02.055692 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.055610    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21064ba7-7cb3-4cbb-8da0-635060293e70-openshift-service-ca\") pod \"21064ba7-7cb3-4cbb-8da0-635060293e70\" (UID: \"21064ba7-7cb3-4cbb-8da0-635060293e70\") "
Aug 02 10:19:02.055692 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.055639    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41523f51-825d-4ac7-abf5-1a38afd5b01c-plugins-config-volume\") pod \"41523f51-825d-4ac7-abf5-1a38afd5b01c\" (UID: \"41523f51-825d-4ac7-abf5-1a38afd5b01c\") "
Aug 02 10:19:02.055692 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.055669    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcvvw\" (UniqueName: \"kubernetes.io/projected/21064ba7-7cb3-4cbb-8da0-635060293e70-kube-api-access-wcvvw\") pod \"21064ba7-7cb3-4cbb-8da0-635060293e70\" (UID: \"21064ba7-7cb3-4cbb-8da0-635060293e70\") "
Aug 02 10:19:02.055853 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.055732    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21064ba7-7cb3-4cbb-8da0-635060293e70-maas-api-tls\") pod \"21064ba7-7cb3-4cbb-8da0-635060293e70\" (UID: \"21064ba7-7cb3-4cbb-8da0-635060293e70\") "
Aug 02 10:19:02.055940 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.055917    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a67f2645-4322-431f-903f-02cdc067e636-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a67f2645-4322-431f-903f-02cdc067e636" (UID: "a67f2645-4322-431f-903f-02cdc067e636"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 02 10:19:02.056017 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.055959    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21064ba7-7cb3-4cbb-8da0-635060293e70-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "21064ba7-7cb3-4cbb-8da0-635060293e70" (UID: "21064ba7-7cb3-4cbb-8da0-635060293e70"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 02 10:19:02.056017 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.055979    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a67f2645-4322-431f-903f-02cdc067e636-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:19:02.056132 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.056035    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41523f51-825d-4ac7-abf5-1a38afd5b01c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "41523f51-825d-4ac7-abf5-1a38afd5b01c" (UID: "41523f51-825d-4ac7-abf5-1a38afd5b01c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 02 10:19:02.057724 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.057697    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a67f2645-4322-431f-903f-02cdc067e636-kube-api-access-q46bq" (OuterVolumeSpecName: "kube-api-access-q46bq") pod "a67f2645-4322-431f-903f-02cdc067e636" (UID: "a67f2645-4322-431f-903f-02cdc067e636"). InnerVolumeSpecName "kube-api-access-q46bq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 02 10:19:02.057828 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.057721    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21064ba7-7cb3-4cbb-8da0-635060293e70-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "21064ba7-7cb3-4cbb-8da0-635060293e70" (UID: "21064ba7-7cb3-4cbb-8da0-635060293e70"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 02 10:19:02.057828 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.057729    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41523f51-825d-4ac7-abf5-1a38afd5b01c-kube-api-access-ds6v6" (OuterVolumeSpecName: "kube-api-access-ds6v6") pod "41523f51-825d-4ac7-abf5-1a38afd5b01c" (UID: "41523f51-825d-4ac7-abf5-1a38afd5b01c"). InnerVolumeSpecName "kube-api-access-ds6v6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 02 10:19:02.057828 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.057763    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21064ba7-7cb3-4cbb-8da0-635060293e70-kube-api-access-wcvvw" (OuterVolumeSpecName: "kube-api-access-wcvvw") pod "21064ba7-7cb3-4cbb-8da0-635060293e70" (UID: "21064ba7-7cb3-4cbb-8da0-635060293e70"). InnerVolumeSpecName "kube-api-access-wcvvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 02 10:19:02.157015 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.156991    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcvvw\" (UniqueName: \"kubernetes.io/projected/21064ba7-7cb3-4cbb-8da0-635060293e70-kube-api-access-wcvvw\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:19:02.157136 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.157017    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21064ba7-7cb3-4cbb-8da0-635060293e70-maas-api-tls\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:19:02.157136 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.157032    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q46bq\" (UniqueName: \"kubernetes.io/projected/a67f2645-4322-431f-903f-02cdc067e636-kube-api-access-q46bq\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:19:02.157136 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.157044    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds6v6\" (UniqueName: \"kubernetes.io/projected/41523f51-825d-4ac7-abf5-1a38afd5b01c-kube-api-access-ds6v6\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:19:02.157136 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.157056    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21064ba7-7cb3-4cbb-8da0-635060293e70-openshift-service-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:19:02.157136 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.157069    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41523f51-825d-4ac7-abf5-1a38afd5b01c-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:19:02.357308 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.357275    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:19:02.710649 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.710618    2564 generic.go:356] "Generic (PLEG): container finished" podID="41523f51-825d-4ac7-abf5-1a38afd5b01c" containerID="a445415933debbedcedb3490f69bfd8c8b73e0e83db92aa90bbf852fd592c620" exitCode=0
Aug 02 10:19:02.711047 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.710669    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e3a61d-68c7cd9cb6-46bbs"
Aug 02 10:19:02.711047 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.710707    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e3a61d-68c7cd9cb6-46bbs" event={"ID":"41523f51-825d-4ac7-abf5-1a38afd5b01c","Type":"ContainerDied","Data":"a445415933debbedcedb3490f69bfd8c8b73e0e83db92aa90bbf852fd592c620"}
Aug 02 10:19:02.711047 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.710788    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e3a61d-68c7cd9cb6-46bbs" event={"ID":"41523f51-825d-4ac7-abf5-1a38afd5b01c","Type":"ContainerDied","Data":"fdcb35439f9ce2f6232e79ff4d4d8f2bc302804abf527ef1df53358419ed18cb"}
Aug 02 10:19:02.711047 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.710816    2564 scope.go:117] "RemoveContainer" containerID="a445415933debbedcedb3490f69bfd8c8b73e0e83db92aa90bbf852fd592c620"
Aug 02 10:19:02.712055 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.712029    2564 generic.go:356] "Generic (PLEG): container finished" podID="21064ba7-7cb3-4cbb-8da0-635060293e70" containerID="b1fb5568bb69ee996159a81494da3f9de9555c7ed26abe6e963bb0cad20b6b27" exitCode=0
Aug 02 10:19:02.712189 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.712059    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3a61d-5b869b769-vfwx7" event={"ID":"21064ba7-7cb3-4cbb-8da0-635060293e70","Type":"ContainerDied","Data":"b1fb5568bb69ee996159a81494da3f9de9555c7ed26abe6e963bb0cad20b6b27"}
Aug 02 10:19:02.712189 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.712071    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3a61d-5b869b769-vfwx7"
Aug 02 10:19:02.712189 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.712104    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3a61d-5b869b769-vfwx7" event={"ID":"21064ba7-7cb3-4cbb-8da0-635060293e70","Type":"ContainerDied","Data":"530f7aa081492cf225a73577cf85f91ce1b4d6ede42509e7f298377110c6d4b4"}
Aug 02 10:19:02.713465 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.713441    2564 generic.go:356] "Generic (PLEG): container finished" podID="a67f2645-4322-431f-903f-02cdc067e636" containerID="400750cd8e3a740e66008443258abdc1dbbcd8ec36fbddb0dc6265e4ceab0975" exitCode=0
Aug 02 10:19:02.713586 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.713488    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3a61d-6677bdb548-qxhpt"
Aug 02 10:19:02.713586 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.713485    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3a61d-6677bdb548-qxhpt" event={"ID":"a67f2645-4322-431f-903f-02cdc067e636","Type":"ContainerDied","Data":"400750cd8e3a740e66008443258abdc1dbbcd8ec36fbddb0dc6265e4ceab0975"}
Aug 02 10:19:02.713702 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.713607    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e3a61d-6677bdb548-qxhpt" event={"ID":"a67f2645-4322-431f-903f-02cdc067e636","Type":"ContainerDied","Data":"669f30599f2f7036f145b3d04a8893f2670a42cfb75423a782c1affc21c490e7"}
Aug 02 10:19:02.719937 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.719922    2564 scope.go:117] "RemoveContainer" containerID="a445415933debbedcedb3490f69bfd8c8b73e0e83db92aa90bbf852fd592c620"
Aug 02 10:19:02.720199 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:19:02.720175    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a445415933debbedcedb3490f69bfd8c8b73e0e83db92aa90bbf852fd592c620\": container with ID starting with a445415933debbedcedb3490f69bfd8c8b73e0e83db92aa90bbf852fd592c620 not found: ID does not exist" containerID="a445415933debbedcedb3490f69bfd8c8b73e0e83db92aa90bbf852fd592c620"
Aug 02 10:19:02.720335 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.720204    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a445415933debbedcedb3490f69bfd8c8b73e0e83db92aa90bbf852fd592c620"} err="failed to get container status \"a445415933debbedcedb3490f69bfd8c8b73e0e83db92aa90bbf852fd592c620\": rpc error: code = NotFound desc = could not find container \"a445415933debbedcedb3490f69bfd8c8b73e0e83db92aa90bbf852fd592c620\": container with ID starting with a445415933debbedcedb3490f69bfd8c8b73e0e83db92aa90bbf852fd592c620 not found: ID does not exist"
Aug 02 10:19:02.720335 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.720220    2564 scope.go:117] "RemoveContainer" containerID="b1fb5568bb69ee996159a81494da3f9de9555c7ed26abe6e963bb0cad20b6b27"
Aug 02 10:19:02.731694 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.731676    2564 scope.go:117] "RemoveContainer" containerID="b1fb5568bb69ee996159a81494da3f9de9555c7ed26abe6e963bb0cad20b6b27"
Aug 02 10:19:02.731918 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:19:02.731900    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1fb5568bb69ee996159a81494da3f9de9555c7ed26abe6e963bb0cad20b6b27\": container with ID starting with b1fb5568bb69ee996159a81494da3f9de9555c7ed26abe6e963bb0cad20b6b27 not found: ID does not exist" containerID="b1fb5568bb69ee996159a81494da3f9de9555c7ed26abe6e963bb0cad20b6b27"
Aug 02 10:19:02.731961 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.731925    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1fb5568bb69ee996159a81494da3f9de9555c7ed26abe6e963bb0cad20b6b27"} err="failed to get container status \"b1fb5568bb69ee996159a81494da3f9de9555c7ed26abe6e963bb0cad20b6b27\": rpc error: code = NotFound desc = could not find container \"b1fb5568bb69ee996159a81494da3f9de9555c7ed26abe6e963bb0cad20b6b27\": container with ID starting with b1fb5568bb69ee996159a81494da3f9de9555c7ed26abe6e963bb0cad20b6b27 not found: ID does not exist"
Aug 02 10:19:02.731961 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.731941    2564 scope.go:117] "RemoveContainer" containerID="400750cd8e3a740e66008443258abdc1dbbcd8ec36fbddb0dc6265e4ceab0975"
Aug 02 10:19:02.738953 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.738936    2564 scope.go:117] "RemoveContainer" containerID="400750cd8e3a740e66008443258abdc1dbbcd8ec36fbddb0dc6265e4ceab0975"
Aug 02 10:19:02.739352 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:19:02.739261    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"400750cd8e3a740e66008443258abdc1dbbcd8ec36fbddb0dc6265e4ceab0975\": container with ID starting with 400750cd8e3a740e66008443258abdc1dbbcd8ec36fbddb0dc6265e4ceab0975 not found: ID does not exist" containerID="400750cd8e3a740e66008443258abdc1dbbcd8ec36fbddb0dc6265e4ceab0975"
Aug 02 10:19:02.739352 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.739296    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"400750cd8e3a740e66008443258abdc1dbbcd8ec36fbddb0dc6265e4ceab0975"} err="failed to get container status \"400750cd8e3a740e66008443258abdc1dbbcd8ec36fbddb0dc6265e4ceab0975\": rpc error: code = NotFound desc = could not find container \"400750cd8e3a740e66008443258abdc1dbbcd8ec36fbddb0dc6265e4ceab0975\": container with ID starting with 400750cd8e3a740e66008443258abdc1dbbcd8ec36fbddb0dc6265e4ceab0975 not found: ID does not exist"
Aug 02 10:19:02.742293 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.742274    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e3a61d-68c7cd9cb6-46bbs"]
Aug 02 10:19:02.749959 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.749941    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e3a61d-68c7cd9cb6-46bbs"]
Aug 02 10:19:02.762418 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.762398    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e3a61d-5b869b769-vfwx7"]
Aug 02 10:19:02.771439 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.771421    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e3a61d-5b869b769-vfwx7"]
Aug 02 10:19:02.787637 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.787617    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e3a61d-6677bdb548-qxhpt"]
Aug 02 10:19:02.797247 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.797211    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e3a61d-6677bdb548-qxhpt"]
Aug 02 10:19:02.880062 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.880033    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21064ba7-7cb3-4cbb-8da0-635060293e70" path="/var/lib/kubelet/pods/21064ba7-7cb3-4cbb-8da0-635060293e70/volumes"
Aug 02 10:19:02.880441 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.880424    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41523f51-825d-4ac7-abf5-1a38afd5b01c" path="/var/lib/kubelet/pods/41523f51-825d-4ac7-abf5-1a38afd5b01c/volumes"
Aug 02 10:19:02.880784 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:02.880768    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a67f2645-4322-431f-903f-02cdc067e636" path="/var/lib/kubelet/pods/a67f2645-4322-431f-903f-02cdc067e636/volumes"
Aug 02 10:19:07.426558 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:07.426526    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:19:13.253949 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:13.253903    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:19:13.546352 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:13.546248    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:19:13.794287 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:13.794253    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:19:24.344787 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.344712    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6c8adc-845494bb57-vb7q8"]
Aug 02 10:19:24.345198 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.344926    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6c8adc-845494bb57-vb7q8" podUID="691f2e6f-fecf-4910-8a08-ed6eecb4e48e" containerName="maas-api" containerID="cri-o://4bd274a3e1ddde0edaf19b0653195d41970ac9ca8e892cee4130597924552f1d" gracePeriod=30
Aug 02 10:19:24.372632 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.372579    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:19:24.394107 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.394083    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6c8adc-5d47ddc54b-44wsx"]
Aug 02 10:19:24.394312 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.394291    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-6c8adc-5d47ddc54b-44wsx" podUID="cab6d9f3-ef7b-4b72-a578-f05013dc2f37" containerName="payload-processing" containerID="cri-o://7be6d3cb76309b33bbb024947df73d1dd0620e786359925b7af1bf5c023c528d" gracePeriod=30
Aug 02 10:19:24.407424 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.407397    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6c8adc-6f7995f884-j4ph8"]
Aug 02 10:19:24.407655 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.407631    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6c8adc-6f7995f884-j4ph8" podUID="d686e6ac-b668-42ca-bf58-7659ba892f11" containerName="payload-pre-processing" containerID="cri-o://606392366c3ac2cbb2d0bce2ffb253d6624e9585f7c391b499bc6cdd862f687e" gracePeriod=30
Aug 02 10:19:24.445106 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.445083    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:19:24.543592 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.543554    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:19:24.644914 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.644883    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:19:24.688759 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.688727    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6c8adc-845494bb57-vb7q8"
Aug 02 10:19:24.691652 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.691629    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6c8adc-5d47ddc54b-44wsx"
Aug 02 10:19:24.705477 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.705460    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6c8adc-6f7995f884-j4ph8"
Aug 02 10:19:24.807405 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.807375    2564 generic.go:356] "Generic (PLEG): container finished" podID="d686e6ac-b668-42ca-bf58-7659ba892f11" containerID="606392366c3ac2cbb2d0bce2ffb253d6624e9585f7c391b499bc6cdd862f687e" exitCode=0
Aug 02 10:19:24.807555 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.807412    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6c8adc-6f7995f884-j4ph8"
Aug 02 10:19:24.807555 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.807459    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6c8adc-6f7995f884-j4ph8" event={"ID":"d686e6ac-b668-42ca-bf58-7659ba892f11","Type":"ContainerDied","Data":"606392366c3ac2cbb2d0bce2ffb253d6624e9585f7c391b499bc6cdd862f687e"}
Aug 02 10:19:24.807555 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.807508    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6c8adc-6f7995f884-j4ph8" event={"ID":"d686e6ac-b668-42ca-bf58-7659ba892f11","Type":"ContainerDied","Data":"a5abbbf96f2b018c9ce3269ed97cb4a4fdc948c8b8d34a972c0fc0c236e04ad1"}
Aug 02 10:19:24.807555 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.807533    2564 scope.go:117] "RemoveContainer" containerID="606392366c3ac2cbb2d0bce2ffb253d6624e9585f7c391b499bc6cdd862f687e"
Aug 02 10:19:24.808684 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.808663    2564 generic.go:356] "Generic (PLEG): container finished" podID="cab6d9f3-ef7b-4b72-a578-f05013dc2f37" containerID="7be6d3cb76309b33bbb024947df73d1dd0620e786359925b7af1bf5c023c528d" exitCode=0
Aug 02 10:19:24.808806 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.808702    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6c8adc-5d47ddc54b-44wsx"
Aug 02 10:19:24.808806 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.808720    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6c8adc-5d47ddc54b-44wsx" event={"ID":"cab6d9f3-ef7b-4b72-a578-f05013dc2f37","Type":"ContainerDied","Data":"7be6d3cb76309b33bbb024947df73d1dd0620e786359925b7af1bf5c023c528d"}
Aug 02 10:19:24.808806 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.808771    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6c8adc-5d47ddc54b-44wsx" event={"ID":"cab6d9f3-ef7b-4b72-a578-f05013dc2f37","Type":"ContainerDied","Data":"be0c6838c7c1d03ba2c2635025b00ccc19e00c18b7fc577301fce6a06594dda3"}
Aug 02 10:19:24.810000 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.809982    2564 generic.go:356] "Generic (PLEG): container finished" podID="691f2e6f-fecf-4910-8a08-ed6eecb4e48e" containerID="4bd274a3e1ddde0edaf19b0653195d41970ac9ca8e892cee4130597924552f1d" exitCode=0
Aug 02 10:19:24.810109 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.810023    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6c8adc-845494bb57-vb7q8"
Aug 02 10:19:24.810109 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.810031    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6c8adc-845494bb57-vb7q8" event={"ID":"691f2e6f-fecf-4910-8a08-ed6eecb4e48e","Type":"ContainerDied","Data":"4bd274a3e1ddde0edaf19b0653195d41970ac9ca8e892cee4130597924552f1d"}
Aug 02 10:19:24.810109 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.810076    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6c8adc-845494bb57-vb7q8" event={"ID":"691f2e6f-fecf-4910-8a08-ed6eecb4e48e","Type":"ContainerDied","Data":"a98bde379c34d0727ec832725bdd4cd19a490f721bd02212b0d9995356252445"}
Aug 02 10:19:24.815377 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.815360    2564 scope.go:117] "RemoveContainer" containerID="606392366c3ac2cbb2d0bce2ffb253d6624e9585f7c391b499bc6cdd862f687e"
Aug 02 10:19:24.815621 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:19:24.815602    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"606392366c3ac2cbb2d0bce2ffb253d6624e9585f7c391b499bc6cdd862f687e\": container with ID starting with 606392366c3ac2cbb2d0bce2ffb253d6624e9585f7c391b499bc6cdd862f687e not found: ID does not exist" containerID="606392366c3ac2cbb2d0bce2ffb253d6624e9585f7c391b499bc6cdd862f687e"
Aug 02 10:19:24.815697 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.815630    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"606392366c3ac2cbb2d0bce2ffb253d6624e9585f7c391b499bc6cdd862f687e"} err="failed to get container status \"606392366c3ac2cbb2d0bce2ffb253d6624e9585f7c391b499bc6cdd862f687e\": rpc error: code = NotFound desc = could not find container \"606392366c3ac2cbb2d0bce2ffb253d6624e9585f7c391b499bc6cdd862f687e\": container with ID starting with 606392366c3ac2cbb2d0bce2ffb253d6624e9585f7c391b499bc6cdd862f687e not found: ID does not exist"
Aug 02 10:19:24.815697 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.815653    2564 scope.go:117] "RemoveContainer" containerID="7be6d3cb76309b33bbb024947df73d1dd0620e786359925b7af1bf5c023c528d"
Aug 02 10:19:24.822800 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.822787    2564 scope.go:117] "RemoveContainer" containerID="7be6d3cb76309b33bbb024947df73d1dd0620e786359925b7af1bf5c023c528d"
Aug 02 10:19:24.823048 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:19:24.823030    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7be6d3cb76309b33bbb024947df73d1dd0620e786359925b7af1bf5c023c528d\": container with ID starting with 7be6d3cb76309b33bbb024947df73d1dd0620e786359925b7af1bf5c023c528d not found: ID does not exist" containerID="7be6d3cb76309b33bbb024947df73d1dd0620e786359925b7af1bf5c023c528d"
Aug 02 10:19:24.823117 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.823056    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7be6d3cb76309b33bbb024947df73d1dd0620e786359925b7af1bf5c023c528d"} err="failed to get container status \"7be6d3cb76309b33bbb024947df73d1dd0620e786359925b7af1bf5c023c528d\": rpc error: code = NotFound desc = could not find container \"7be6d3cb76309b33bbb024947df73d1dd0620e786359925b7af1bf5c023c528d\": container with ID starting with 7be6d3cb76309b33bbb024947df73d1dd0620e786359925b7af1bf5c023c528d not found: ID does not exist"
Aug 02 10:19:24.823117 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.823078    2564 scope.go:117] "RemoveContainer" containerID="4bd274a3e1ddde0edaf19b0653195d41970ac9ca8e892cee4130597924552f1d"
Aug 02 10:19:24.829796 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.829779    2564 scope.go:117] "RemoveContainer" containerID="4bd274a3e1ddde0edaf19b0653195d41970ac9ca8e892cee4130597924552f1d"
Aug 02 10:19:24.830024 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:19:24.830006    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bd274a3e1ddde0edaf19b0653195d41970ac9ca8e892cee4130597924552f1d\": container with ID starting with 4bd274a3e1ddde0edaf19b0653195d41970ac9ca8e892cee4130597924552f1d not found: ID does not exist" containerID="4bd274a3e1ddde0edaf19b0653195d41970ac9ca8e892cee4130597924552f1d"
Aug 02 10:19:24.830075 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.830028    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bd274a3e1ddde0edaf19b0653195d41970ac9ca8e892cee4130597924552f1d"} err="failed to get container status \"4bd274a3e1ddde0edaf19b0653195d41970ac9ca8e892cee4130597924552f1d\": rpc error: code = NotFound desc = could not find container \"4bd274a3e1ddde0edaf19b0653195d41970ac9ca8e892cee4130597924552f1d\": container with ID starting with 4bd274a3e1ddde0edaf19b0653195d41970ac9ca8e892cee4130597924552f1d not found: ID does not exist"
Aug 02 10:19:24.838345 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.838328    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qn55c\" (UniqueName: \"kubernetes.io/projected/d686e6ac-b668-42ca-bf58-7659ba892f11-kube-api-access-qn55c\") pod \"d686e6ac-b668-42ca-bf58-7659ba892f11\" (UID: \"d686e6ac-b668-42ca-bf58-7659ba892f11\") "
Aug 02 10:19:24.838420 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.838367    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/691f2e6f-fecf-4910-8a08-ed6eecb4e48e-maas-api-tls\") pod \"691f2e6f-fecf-4910-8a08-ed6eecb4e48e\" (UID: \"691f2e6f-fecf-4910-8a08-ed6eecb4e48e\") "
Aug 02 10:19:24.838420 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.838393    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/691f2e6f-fecf-4910-8a08-ed6eecb4e48e-openshift-service-ca\") pod \"691f2e6f-fecf-4910-8a08-ed6eecb4e48e\" (UID: \"691f2e6f-fecf-4910-8a08-ed6eecb4e48e\") "
Aug 02 10:19:24.838420 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.838409    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d686e6ac-b668-42ca-bf58-7659ba892f11-plugins-config-volume\") pod \"d686e6ac-b668-42ca-bf58-7659ba892f11\" (UID: \"d686e6ac-b668-42ca-bf58-7659ba892f11\") "
Aug 02 10:19:24.838568 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.838443    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cab6d9f3-ef7b-4b72-a578-f05013dc2f37-plugins-config-volume\") pod \"cab6d9f3-ef7b-4b72-a578-f05013dc2f37\" (UID: \"cab6d9f3-ef7b-4b72-a578-f05013dc2f37\") "
Aug 02 10:19:24.838568 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.838495    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hkbmh\" (UniqueName: \"kubernetes.io/projected/691f2e6f-fecf-4910-8a08-ed6eecb4e48e-kube-api-access-hkbmh\") pod \"691f2e6f-fecf-4910-8a08-ed6eecb4e48e\" (UID: \"691f2e6f-fecf-4910-8a08-ed6eecb4e48e\") "
Aug 02 10:19:24.838568 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.838527    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w65kg\" (UniqueName: \"kubernetes.io/projected/cab6d9f3-ef7b-4b72-a578-f05013dc2f37-kube-api-access-w65kg\") pod \"cab6d9f3-ef7b-4b72-a578-f05013dc2f37\" (UID: \"cab6d9f3-ef7b-4b72-a578-f05013dc2f37\") "
Aug 02 10:19:24.838849 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.838801    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d686e6ac-b668-42ca-bf58-7659ba892f11-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d686e6ac-b668-42ca-bf58-7659ba892f11" (UID: "d686e6ac-b668-42ca-bf58-7659ba892f11"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 02 10:19:24.838849 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.838801    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/691f2e6f-fecf-4910-8a08-ed6eecb4e48e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "691f2e6f-fecf-4910-8a08-ed6eecb4e48e" (UID: "691f2e6f-fecf-4910-8a08-ed6eecb4e48e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 02 10:19:24.839033 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.838883    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cab6d9f3-ef7b-4b72-a578-f05013dc2f37-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cab6d9f3-ef7b-4b72-a578-f05013dc2f37" (UID: "cab6d9f3-ef7b-4b72-a578-f05013dc2f37"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 02 10:19:24.840373 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.840343    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/691f2e6f-fecf-4910-8a08-ed6eecb4e48e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "691f2e6f-fecf-4910-8a08-ed6eecb4e48e" (UID: "691f2e6f-fecf-4910-8a08-ed6eecb4e48e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 02 10:19:24.840472 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.840453    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d686e6ac-b668-42ca-bf58-7659ba892f11-kube-api-access-qn55c" (OuterVolumeSpecName: "kube-api-access-qn55c") pod "d686e6ac-b668-42ca-bf58-7659ba892f11" (UID: "d686e6ac-b668-42ca-bf58-7659ba892f11"). InnerVolumeSpecName "kube-api-access-qn55c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 02 10:19:24.840612 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.840589    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/691f2e6f-fecf-4910-8a08-ed6eecb4e48e-kube-api-access-hkbmh" (OuterVolumeSpecName: "kube-api-access-hkbmh") pod "691f2e6f-fecf-4910-8a08-ed6eecb4e48e" (UID: "691f2e6f-fecf-4910-8a08-ed6eecb4e48e"). InnerVolumeSpecName "kube-api-access-hkbmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 02 10:19:24.840686 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.840623    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cab6d9f3-ef7b-4b72-a578-f05013dc2f37-kube-api-access-w65kg" (OuterVolumeSpecName: "kube-api-access-w65kg") pod "cab6d9f3-ef7b-4b72-a578-f05013dc2f37" (UID: "cab6d9f3-ef7b-4b72-a578-f05013dc2f37"). InnerVolumeSpecName "kube-api-access-w65kg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 02 10:19:24.938046 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.938010    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:19:24.940033 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.940014    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkbmh\" (UniqueName: \"kubernetes.io/projected/691f2e6f-fecf-4910-8a08-ed6eecb4e48e-kube-api-access-hkbmh\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:19:24.940084 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.940036    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w65kg\" (UniqueName: \"kubernetes.io/projected/cab6d9f3-ef7b-4b72-a578-f05013dc2f37-kube-api-access-w65kg\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:19:24.940084 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.940046    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn55c\" (UniqueName: \"kubernetes.io/projected/d686e6ac-b668-42ca-bf58-7659ba892f11-kube-api-access-qn55c\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:19:24.940084 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.940056    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/691f2e6f-fecf-4910-8a08-ed6eecb4e48e-maas-api-tls\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:19:24.940084 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.940064    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/691f2e6f-fecf-4910-8a08-ed6eecb4e48e-openshift-service-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:19:24.940084 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.940073    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d686e6ac-b668-42ca-bf58-7659ba892f11-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:19:24.940084 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:24.940082    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cab6d9f3-ef7b-4b72-a578-f05013dc2f37-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:19:25.124971 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:25.124942    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6c8adc-6f7995f884-j4ph8"]
Aug 02 10:19:25.129856 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:25.129833    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6c8adc-6f7995f884-j4ph8"]
Aug 02 10:19:25.142961 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:25.142908    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6c8adc-5d47ddc54b-44wsx"]
Aug 02 10:19:25.146887 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:25.146866    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6c8adc-5d47ddc54b-44wsx"]
Aug 02 10:19:25.156976 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:25.156959    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6c8adc-845494bb57-vb7q8"]
Aug 02 10:19:25.159389 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:25.159371    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6c8adc-845494bb57-vb7q8"]
Aug 02 10:19:26.885958 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:26.885923    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="691f2e6f-fecf-4910-8a08-ed6eecb4e48e" path="/var/lib/kubelet/pods/691f2e6f-fecf-4910-8a08-ed6eecb4e48e/volumes"
Aug 02 10:19:26.886368 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:26.886304    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cab6d9f3-ef7b-4b72-a578-f05013dc2f37" path="/var/lib/kubelet/pods/cab6d9f3-ef7b-4b72-a578-f05013dc2f37/volumes"
Aug 02 10:19:26.886631 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:26.886616    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d686e6ac-b668-42ca-bf58-7659ba892f11" path="/var/lib/kubelet/pods/d686e6ac-b668-42ca-bf58-7659ba892f11/volumes"
Aug 02 10:19:30.914568 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:30.914533    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:19:30.990457 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:30.990426    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:19:31.235473 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:31.235392    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:19:31.512256 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:31.512149    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:19:31.827966 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:31.827877    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:19:32.227783 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:32.227750    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:19:32.558420 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:32.558329    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:19:35.386729 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:35.386692    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:19:35.477107 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:35.477074    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:19:36.605217 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:36.605180    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-931e8a-68c4d5c44c-8vj5v"]
Aug 02 10:19:36.605612 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:36.605488    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21064ba7-7cb3-4cbb-8da0-635060293e70" containerName="maas-api"
Aug 02 10:19:36.605612 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:36.605499    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="21064ba7-7cb3-4cbb-8da0-635060293e70" containerName="maas-api"
Aug 02 10:19:36.605612 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:36.605506    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cab6d9f3-ef7b-4b72-a578-f05013dc2f37" containerName="payload-processing"
Aug 02 10:19:36.605612 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:36.605512    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cab6d9f3-ef7b-4b72-a578-f05013dc2f37" containerName="payload-processing"
Aug 02 10:19:36.605612 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:36.605523    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d686e6ac-b668-42ca-bf58-7659ba892f11" containerName="payload-pre-processing"
Aug 02 10:19:36.605612 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:36.605530    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d686e6ac-b668-42ca-bf58-7659ba892f11" containerName="payload-pre-processing"
Aug 02 10:19:36.605612 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:36.605536    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41523f51-825d-4ac7-abf5-1a38afd5b01c" containerName="payload-processing"
Aug 02 10:19:36.605612 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:36.605542    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="41523f51-825d-4ac7-abf5-1a38afd5b01c" containerName="payload-processing"
Aug 02 10:19:36.605612 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:36.605553    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a67f2645-4322-431f-903f-02cdc067e636" containerName="payload-pre-processing"
Aug 02 10:19:36.605612 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:36.605558    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a67f2645-4322-431f-903f-02cdc067e636" containerName="payload-pre-processing"
Aug 02 10:19:36.605612 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:36.605569    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="691f2e6f-fecf-4910-8a08-ed6eecb4e48e" containerName="maas-api"
Aug 02 10:19:36.605612 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:36.605575    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="691f2e6f-fecf-4910-8a08-ed6eecb4e48e" containerName="maas-api"
Aug 02 10:19:36.606029 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:36.605628    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="41523f51-825d-4ac7-abf5-1a38afd5b01c" containerName="payload-processing"
Aug 02 10:19:36.606029 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:36.605636    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a67f2645-4322-431f-903f-02cdc067e636" containerName="payload-pre-processing"
Aug 02 10:19:36.606029 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:36.605642    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="21064ba7-7cb3-4cbb-8da0-635060293e70" containerName="maas-api"
Aug 02 10:19:36.606029 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:36.605647    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cab6d9f3-ef7b-4b72-a578-f05013dc2f37" containerName="payload-processing"
Aug 02 10:19:36.606029 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:36.605652    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="691f2e6f-fecf-4910-8a08-ed6eecb4e48e" containerName="maas-api"
Aug 02 10:19:36.606029 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:36.605658    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d686e6ac-b668-42ca-bf58-7659ba892f11" containerName="payload-pre-processing"
Aug 02 10:19:36.608449 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:36.608430    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-931e8a-68c4d5c44c-8vj5v"
Aug 02 10:19:36.610684 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:36.610659    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-931e8a-serving-cert\""
Aug 02 10:19:36.620776 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:36.620754    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-931e8a-68c4d5c44c-8vj5v"]
Aug 02 10:19:36.638601 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:36.638574    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:19:36.730497 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:36.730465    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p7jb\" (UniqueName: \"kubernetes.io/projected/1877b800-7c5f-4298-8eb2-b45189118c2a-kube-api-access-7p7jb\") pod \"maas-api-e2e-api-a-931e8a-68c4d5c44c-8vj5v\" (UID: \"1877b800-7c5f-4298-8eb2-b45189118c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-931e8a-68c4d5c44c-8vj5v"
Aug 02 10:19:36.730641 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:36.730521    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1877b800-7c5f-4298-8eb2-b45189118c2a-maas-api-tls\") pod \"maas-api-e2e-api-a-931e8a-68c4d5c44c-8vj5v\" (UID: \"1877b800-7c5f-4298-8eb2-b45189118c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-931e8a-68c4d5c44c-8vj5v"
Aug 02 10:19:36.730704 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:36.730626    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1877b800-7c5f-4298-8eb2-b45189118c2a-openshift-service-ca\") pod \"maas-api-e2e-api-a-931e8a-68c4d5c44c-8vj5v\" (UID: \"1877b800-7c5f-4298-8eb2-b45189118c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-931e8a-68c4d5c44c-8vj5v"
Aug 02 10:19:36.831578 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:36.831541    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1877b800-7c5f-4298-8eb2-b45189118c2a-openshift-service-ca\") pod \"maas-api-e2e-api-a-931e8a-68c4d5c44c-8vj5v\" (UID: \"1877b800-7c5f-4298-8eb2-b45189118c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-931e8a-68c4d5c44c-8vj5v"
Aug 02 10:19:36.831721 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:36.831630    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p7jb\" (UniqueName: \"kubernetes.io/projected/1877b800-7c5f-4298-8eb2-b45189118c2a-kube-api-access-7p7jb\") pod \"maas-api-e2e-api-a-931e8a-68c4d5c44c-8vj5v\" (UID: \"1877b800-7c5f-4298-8eb2-b45189118c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-931e8a-68c4d5c44c-8vj5v"
Aug 02 10:19:36.831721 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:36.831669    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1877b800-7c5f-4298-8eb2-b45189118c2a-maas-api-tls\") pod \"maas-api-e2e-api-a-931e8a-68c4d5c44c-8vj5v\" (UID: \"1877b800-7c5f-4298-8eb2-b45189118c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-931e8a-68c4d5c44c-8vj5v"
Aug 02 10:19:36.831803 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:19:36.831783    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-931e8a-serving-cert: secret "maas-api-e2e-api-a-931e8a-serving-cert" not found
Aug 02 10:19:36.831868 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:19:36.831857    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1877b800-7c5f-4298-8eb2-b45189118c2a-maas-api-tls podName:1877b800-7c5f-4298-8eb2-b45189118c2a nodeName:}" failed. No retries permitted until 2026-08-02 10:19:37.331833953 +0000 UTC m=+2801.046639381 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1877b800-7c5f-4298-8eb2-b45189118c2a-maas-api-tls") pod "maas-api-e2e-api-a-931e8a-68c4d5c44c-8vj5v" (UID: "1877b800-7c5f-4298-8eb2-b45189118c2a") : secret "maas-api-e2e-api-a-931e8a-serving-cert" not found
Aug 02 10:19:36.832218 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:36.832200    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1877b800-7c5f-4298-8eb2-b45189118c2a-openshift-service-ca\") pod \"maas-api-e2e-api-a-931e8a-68c4d5c44c-8vj5v\" (UID: \"1877b800-7c5f-4298-8eb2-b45189118c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-931e8a-68c4d5c44c-8vj5v"
Aug 02 10:19:36.841427 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:36.841406    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p7jb\" (UniqueName: \"kubernetes.io/projected/1877b800-7c5f-4298-8eb2-b45189118c2a-kube-api-access-7p7jb\") pod \"maas-api-e2e-api-a-931e8a-68c4d5c44c-8vj5v\" (UID: \"1877b800-7c5f-4298-8eb2-b45189118c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-931e8a-68c4d5c44c-8vj5v"
Aug 02 10:19:36.969920 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:36.969866    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:19:37.125643 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:37.125611    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:19:37.220482 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:37.220401    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:19:37.246286 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:37.246260    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-931e8a-6986bf6c84-d5zsn"]
Aug 02 10:19:37.249604 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:37.249590    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-931e8a-6986bf6c84-d5zsn"
Aug 02 10:19:37.252261 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:37.252227    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-931e8a\""
Aug 02 10:19:37.262008 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:37.261985    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-931e8a-6986bf6c84-d5zsn"]
Aug 02 10:19:37.272747 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:37.272722    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-931e8a-6fb9976b98-bc8hp"]
Aug 02 10:19:37.275636 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:37.275617    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-931e8a-6fb9976b98-bc8hp"
Aug 02 10:19:37.280005 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:37.279986    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-931e8a-dockercfg-pbplc\""
Aug 02 10:19:37.290089 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:37.290068    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-931e8a-6fb9976b98-bc8hp"]
Aug 02 10:19:37.336425 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:37.336397    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1877b800-7c5f-4298-8eb2-b45189118c2a-maas-api-tls\") pod \"maas-api-e2e-api-a-931e8a-68c4d5c44c-8vj5v\" (UID: \"1877b800-7c5f-4298-8eb2-b45189118c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-931e8a-68c4d5c44c-8vj5v"
Aug 02 10:19:37.338619 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:37.338603    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1877b800-7c5f-4298-8eb2-b45189118c2a-maas-api-tls\") pod \"maas-api-e2e-api-a-931e8a-68c4d5c44c-8vj5v\" (UID: \"1877b800-7c5f-4298-8eb2-b45189118c2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-931e8a-68c4d5c44c-8vj5v"
Aug 02 10:19:37.437642 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:37.437609    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3690f42d-b0f6-4558-a7de-a5cfa3df7cb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-931e8a-6986bf6c84-d5zsn\" (UID: \"3690f42d-b0f6-4558-a7de-a5cfa3df7cb8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-931e8a-6986bf6c84-d5zsn"
Aug 02 10:19:37.437783 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:37.437652    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qwn2\" (UniqueName: \"kubernetes.io/projected/c8479024-c2a6-4b33-b403-472e80654ebe-kube-api-access-6qwn2\") pod \"payload-processing-e2e-api-a-931e8a-6fb9976b98-bc8hp\" (UID: \"c8479024-c2a6-4b33-b403-472e80654ebe\") " pod="openshift-ingress/payload-processing-e2e-api-a-931e8a-6fb9976b98-bc8hp"
Aug 02 10:19:37.437783 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:37.437724    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8479024-c2a6-4b33-b403-472e80654ebe-plugins-config-volume\") pod \"payload-processing-e2e-api-a-931e8a-6fb9976b98-bc8hp\" (UID: \"c8479024-c2a6-4b33-b403-472e80654ebe\") " pod="openshift-ingress/payload-processing-e2e-api-a-931e8a-6fb9976b98-bc8hp"
Aug 02 10:19:37.437862 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:37.437790    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq6t7\" (UniqueName: \"kubernetes.io/projected/3690f42d-b0f6-4558-a7de-a5cfa3df7cb8-kube-api-access-mq6t7\") pod \"payload-pre-processing-e2e-api-a-931e8a-6986bf6c84-d5zsn\" (UID: \"3690f42d-b0f6-4558-a7de-a5cfa3df7cb8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-931e8a-6986bf6c84-d5zsn"
Aug 02 10:19:37.518892 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:37.518831    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-931e8a-68c4d5c44c-8vj5v"
Aug 02 10:19:37.539106 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:37.539073    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3690f42d-b0f6-4558-a7de-a5cfa3df7cb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-931e8a-6986bf6c84-d5zsn\" (UID: \"3690f42d-b0f6-4558-a7de-a5cfa3df7cb8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-931e8a-6986bf6c84-d5zsn"
Aug 02 10:19:37.539268 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:37.539133    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qwn2\" (UniqueName: \"kubernetes.io/projected/c8479024-c2a6-4b33-b403-472e80654ebe-kube-api-access-6qwn2\") pod \"payload-processing-e2e-api-a-931e8a-6fb9976b98-bc8hp\" (UID: \"c8479024-c2a6-4b33-b403-472e80654ebe\") " pod="openshift-ingress/payload-processing-e2e-api-a-931e8a-6fb9976b98-bc8hp"
Aug 02 10:19:37.539268 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:37.539177    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8479024-c2a6-4b33-b403-472e80654ebe-plugins-config-volume\") pod \"payload-processing-e2e-api-a-931e8a-6fb9976b98-bc8hp\" (UID: \"c8479024-c2a6-4b33-b403-472e80654ebe\") " pod="openshift-ingress/payload-processing-e2e-api-a-931e8a-6fb9976b98-bc8hp"
Aug 02 10:19:37.539395 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:37.539266    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq6t7\" (UniqueName: \"kubernetes.io/projected/3690f42d-b0f6-4558-a7de-a5cfa3df7cb8-kube-api-access-mq6t7\") pod \"payload-pre-processing-e2e-api-a-931e8a-6986bf6c84-d5zsn\" (UID: \"3690f42d-b0f6-4558-a7de-a5cfa3df7cb8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-931e8a-6986bf6c84-d5zsn"
Aug 02 10:19:37.539947 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:37.539922    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3690f42d-b0f6-4558-a7de-a5cfa3df7cb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-931e8a-6986bf6c84-d5zsn\" (UID: \"3690f42d-b0f6-4558-a7de-a5cfa3df7cb8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-931e8a-6986bf6c84-d5zsn"
Aug 02 10:19:37.540066 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:37.539991    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8479024-c2a6-4b33-b403-472e80654ebe-plugins-config-volume\") pod \"payload-processing-e2e-api-a-931e8a-6fb9976b98-bc8hp\" (UID: \"c8479024-c2a6-4b33-b403-472e80654ebe\") " pod="openshift-ingress/payload-processing-e2e-api-a-931e8a-6fb9976b98-bc8hp"
Aug 02 10:19:37.548402 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:37.548372    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq6t7\" (UniqueName: \"kubernetes.io/projected/3690f42d-b0f6-4558-a7de-a5cfa3df7cb8-kube-api-access-mq6t7\") pod \"payload-pre-processing-e2e-api-a-931e8a-6986bf6c84-d5zsn\" (UID: \"3690f42d-b0f6-4558-a7de-a5cfa3df7cb8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-931e8a-6986bf6c84-d5zsn"
Aug 02 10:19:37.548599 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:37.548581    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qwn2\" (UniqueName: \"kubernetes.io/projected/c8479024-c2a6-4b33-b403-472e80654ebe-kube-api-access-6qwn2\") pod \"payload-processing-e2e-api-a-931e8a-6fb9976b98-bc8hp\" (UID: \"c8479024-c2a6-4b33-b403-472e80654ebe\") " pod="openshift-ingress/payload-processing-e2e-api-a-931e8a-6fb9976b98-bc8hp"
Aug 02 10:19:37.559817 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:37.559782    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-931e8a-6986bf6c84-d5zsn"
Aug 02 10:19:37.585735 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:37.585703    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-931e8a-6fb9976b98-bc8hp"
Aug 02 10:19:37.667264 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:37.667213    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-931e8a-68c4d5c44c-8vj5v"]
Aug 02 10:19:37.671010 ip-10-0-141-232 kubenswrapper[2564]: W0802 10:19:37.670978    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1877b800_7c5f_4298_8eb2_b45189118c2a.slice/crio-40d59d86a30dfbe22a297e79b67bfd75c575a688acb052bc7987d95cdb408bb9 WatchSource:0}: Error finding container 40d59d86a30dfbe22a297e79b67bfd75c575a688acb052bc7987d95cdb408bb9: Status 404 returned error can't find the container with id 40d59d86a30dfbe22a297e79b67bfd75c575a688acb052bc7987d95cdb408bb9
Aug 02 10:19:37.718035 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:37.718011    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-931e8a-6986bf6c84-d5zsn"]
Aug 02 10:19:37.720767 ip-10-0-141-232 kubenswrapper[2564]: W0802 10:19:37.720740    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3690f42d_b0f6_4558_a7de_a5cfa3df7cb8.slice/crio-d5c5d8c99f8fe43bcda2016349e76d3210b6273f73df08a4cdd35d433776708f WatchSource:0}: Error finding container d5c5d8c99f8fe43bcda2016349e76d3210b6273f73df08a4cdd35d433776708f: Status 404 returned error can't find the container with id d5c5d8c99f8fe43bcda2016349e76d3210b6273f73df08a4cdd35d433776708f
Aug 02 10:19:37.739377 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:37.739353    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-931e8a-6fb9976b98-bc8hp"]
Aug 02 10:19:37.743276 ip-10-0-141-232 kubenswrapper[2564]: W0802 10:19:37.743249    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8479024_c2a6_4b33_b403_472e80654ebe.slice/crio-2979e4e35f0f035db9851d2f779bb56faf14a322db28aeeb60879932ed92e1ff WatchSource:0}: Error finding container 2979e4e35f0f035db9851d2f779bb56faf14a322db28aeeb60879932ed92e1ff: Status 404 returned error can't find the container with id 2979e4e35f0f035db9851d2f779bb56faf14a322db28aeeb60879932ed92e1ff
Aug 02 10:19:37.872388 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:37.872350    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-931e8a-68c4d5c44c-8vj5v" event={"ID":"1877b800-7c5f-4298-8eb2-b45189118c2a","Type":"ContainerStarted","Data":"40d59d86a30dfbe22a297e79b67bfd75c575a688acb052bc7987d95cdb408bb9"}
Aug 02 10:19:37.873885 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:37.873861    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-931e8a-6fb9976b98-bc8hp" event={"ID":"c8479024-c2a6-4b33-b403-472e80654ebe","Type":"ContainerStarted","Data":"01db110f68d7d6c9e4ef16fd1886d7cfd10a9243667ecba5c3c916b07f66d992"}
Aug 02 10:19:37.874027 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:37.873902    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-931e8a-6fb9976b98-bc8hp" event={"ID":"c8479024-c2a6-4b33-b403-472e80654ebe","Type":"ContainerStarted","Data":"2979e4e35f0f035db9851d2f779bb56faf14a322db28aeeb60879932ed92e1ff"}
Aug 02 10:19:37.874027 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:37.873942    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-931e8a-6fb9976b98-bc8hp"
Aug 02 10:19:37.875435 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:37.875414    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-931e8a-6986bf6c84-d5zsn" event={"ID":"3690f42d-b0f6-4558-a7de-a5cfa3df7cb8","Type":"ContainerStarted","Data":"8eb031958454b6269c57632947e66b204694756816a10246e823e4ee784d55be"}
Aug 02 10:19:37.875536 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:37.875456    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-931e8a-6986bf6c84-d5zsn" event={"ID":"3690f42d-b0f6-4558-a7de-a5cfa3df7cb8","Type":"ContainerStarted","Data":"d5c5d8c99f8fe43bcda2016349e76d3210b6273f73df08a4cdd35d433776708f"}
Aug 02 10:19:37.875536 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:37.875488    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-931e8a-6986bf6c84-d5zsn"
Aug 02 10:19:37.891751 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:37.891710    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-931e8a-6fb9976b98-bc8hp" podStartSLOduration=0.891696537 podStartE2EDuration="891.696537ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:19:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:19:37.890003431 +0000 UTC m=+2801.604808880" watchObservedRunningTime="2026-08-02 10:19:37.891696537 +0000 UTC m=+2801.606501985"
Aug 02 10:19:37.906875 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:37.906832    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-931e8a-6986bf6c84-d5zsn" podStartSLOduration=0.906819953 podStartE2EDuration="906.819953ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:19:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:19:37.905899381 +0000 UTC m=+2801.620704830" watchObservedRunningTime="2026-08-02 10:19:37.906819953 +0000 UTC m=+2801.621625401"
Aug 02 10:19:38.881197 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:38.881163    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-931e8a-68c4d5c44c-8vj5v" event={"ID":"1877b800-7c5f-4298-8eb2-b45189118c2a","Type":"ContainerStarted","Data":"56ba67c9179fcd6cca1c2c45565822c51336f038d52f5ef325aa3def591b3bfc"}
Aug 02 10:19:38.881646 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:38.881323    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-931e8a-68c4d5c44c-8vj5v"
Aug 02 10:19:38.900169 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:38.900122    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-931e8a-68c4d5c44c-8vj5v" podStartSLOduration=2.534209755 podStartE2EDuration="2.900107727s" totalImagesPullingTime="365.897972ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:19:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:19:38.898904198 +0000 UTC m=+2802.613709643" watchObservedRunningTime="2026-08-02 10:19:38.900107727 +0000 UTC m=+2802.614913175"
Aug 02 10:19:44.890567 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:44.890538    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-931e8a-68c4d5c44c-8vj5v"
Aug 02 10:19:48.883317 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:48.883289    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-931e8a-6fb9976b98-bc8hp"
Aug 02 10:19:48.885650 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:19:48.883357    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-931e8a-6986bf6c84-d5zsn"
Aug 02 10:20:00.367807 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:00.367768    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:20:00.849191 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:00.849144    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:20:01.220786 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:01.220742    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:20:01.501418 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:01.501323    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:20:04.548887 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:04.548841    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:20:04.639422 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:04.639383    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:20:05.646834 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:05.646804    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2ea423-9d7c759d4-tbp7x"]
Aug 02 10:20:05.656202 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:05.655523    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ea423-9d7c759d4-tbp7x"
Aug 02 10:20:05.658761 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:05.658740    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-2ea423-serving-cert\""
Aug 02 10:20:05.661364 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:05.661341    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2ea423-9d7c759d4-tbp7x"]
Aug 02 10:20:05.697778 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:05.697746    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:20:05.753017 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:05.752979    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea557ed5-fdec-41bb-9122-ed2529c8890f-maas-api-tls\") pod \"maas-api-e2e-api-b-2ea423-9d7c759d4-tbp7x\" (UID: \"ea557ed5-fdec-41bb-9122-ed2529c8890f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ea423-9d7c759d4-tbp7x"
Aug 02 10:20:05.753214 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:05.753037    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb5kj\" (UniqueName: \"kubernetes.io/projected/ea557ed5-fdec-41bb-9122-ed2529c8890f-kube-api-access-kb5kj\") pod \"maas-api-e2e-api-b-2ea423-9d7c759d4-tbp7x\" (UID: \"ea557ed5-fdec-41bb-9122-ed2529c8890f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ea423-9d7c759d4-tbp7x"
Aug 02 10:20:05.753214 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:05.753071    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea557ed5-fdec-41bb-9122-ed2529c8890f-openshift-service-ca\") pod \"maas-api-e2e-api-b-2ea423-9d7c759d4-tbp7x\" (UID: \"ea557ed5-fdec-41bb-9122-ed2529c8890f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ea423-9d7c759d4-tbp7x"
Aug 02 10:20:05.854214 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:05.854182    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb5kj\" (UniqueName: \"kubernetes.io/projected/ea557ed5-fdec-41bb-9122-ed2529c8890f-kube-api-access-kb5kj\") pod \"maas-api-e2e-api-b-2ea423-9d7c759d4-tbp7x\" (UID: \"ea557ed5-fdec-41bb-9122-ed2529c8890f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ea423-9d7c759d4-tbp7x"
Aug 02 10:20:05.854381 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:05.854219    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea557ed5-fdec-41bb-9122-ed2529c8890f-openshift-service-ca\") pod \"maas-api-e2e-api-b-2ea423-9d7c759d4-tbp7x\" (UID: \"ea557ed5-fdec-41bb-9122-ed2529c8890f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ea423-9d7c759d4-tbp7x"
Aug 02 10:20:05.854381 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:05.854316    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea557ed5-fdec-41bb-9122-ed2529c8890f-maas-api-tls\") pod \"maas-api-e2e-api-b-2ea423-9d7c759d4-tbp7x\" (UID: \"ea557ed5-fdec-41bb-9122-ed2529c8890f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ea423-9d7c759d4-tbp7x"
Aug 02 10:20:05.854456 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:20:05.854413    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-2ea423-serving-cert: secret "maas-api-e2e-api-b-2ea423-serving-cert" not found
Aug 02 10:20:05.854492 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:20:05.854479    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea557ed5-fdec-41bb-9122-ed2529c8890f-maas-api-tls podName:ea557ed5-fdec-41bb-9122-ed2529c8890f nodeName:}" failed. No retries permitted until 2026-08-02 10:20:06.354458884 +0000 UTC m=+2830.069264310 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ea557ed5-fdec-41bb-9122-ed2529c8890f-maas-api-tls") pod "maas-api-e2e-api-b-2ea423-9d7c759d4-tbp7x" (UID: "ea557ed5-fdec-41bb-9122-ed2529c8890f") : secret "maas-api-e2e-api-b-2ea423-serving-cert" not found
Aug 02 10:20:05.854994 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:05.854974    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea557ed5-fdec-41bb-9122-ed2529c8890f-openshift-service-ca\") pod \"maas-api-e2e-api-b-2ea423-9d7c759d4-tbp7x\" (UID: \"ea557ed5-fdec-41bb-9122-ed2529c8890f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ea423-9d7c759d4-tbp7x"
Aug 02 10:20:05.865671 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:05.865642    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb5kj\" (UniqueName: \"kubernetes.io/projected/ea557ed5-fdec-41bb-9122-ed2529c8890f-kube-api-access-kb5kj\") pod \"maas-api-e2e-api-b-2ea423-9d7c759d4-tbp7x\" (UID: \"ea557ed5-fdec-41bb-9122-ed2529c8890f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ea423-9d7c759d4-tbp7x"
Aug 02 10:20:05.936951 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:05.936882    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:20:06.358563 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:06.358487    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea557ed5-fdec-41bb-9122-ed2529c8890f-maas-api-tls\") pod \"maas-api-e2e-api-b-2ea423-9d7c759d4-tbp7x\" (UID: \"ea557ed5-fdec-41bb-9122-ed2529c8890f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ea423-9d7c759d4-tbp7x"
Aug 02 10:20:06.360966 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:06.360935    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea557ed5-fdec-41bb-9122-ed2529c8890f-maas-api-tls\") pod \"maas-api-e2e-api-b-2ea423-9d7c759d4-tbp7x\" (UID: \"ea557ed5-fdec-41bb-9122-ed2529c8890f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ea423-9d7c759d4-tbp7x"
Aug 02 10:20:06.410147 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:06.410123    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:20:06.508114 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:06.508091    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:20:06.527602 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:06.527571    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2ea423-7fb47f76-thdqn"]
Aug 02 10:20:06.531078 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:06.531063    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ea423-7fb47f76-thdqn"
Aug 02 10:20:06.534012 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:06.533989    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-2ea423\""
Aug 02 10:20:06.544014 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:06.543992    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2ea423-7fb47f76-thdqn"]
Aug 02 10:20:06.553016 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:06.552995    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2ea423-57c64b48d-hb24b"]
Aug 02 10:20:06.556042 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:06.556025    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2ea423-57c64b48d-hb24b"
Aug 02 10:20:06.558790 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:06.558765    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-2ea423-dockercfg-kwdvw\""
Aug 02 10:20:06.569425 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:06.569403    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ea423-9d7c759d4-tbp7x"
Aug 02 10:20:06.570524 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:06.570502    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2ea423-57c64b48d-hb24b"]
Aug 02 10:20:06.660602 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:06.660523    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpc5l\" (UniqueName: \"kubernetes.io/projected/d15fe25b-5cd2-4039-8b22-916c89379b75-kube-api-access-jpc5l\") pod \"payload-pre-processing-e2e-api-b-2ea423-7fb47f76-thdqn\" (UID: \"d15fe25b-5cd2-4039-8b22-916c89379b75\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ea423-7fb47f76-thdqn"
Aug 02 10:20:06.661141 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:06.660643    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13837f5d-7f7d-402a-9d49-9bc8370cf97b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2ea423-57c64b48d-hb24b\" (UID: \"13837f5d-7f7d-402a-9d49-9bc8370cf97b\") " pod="openshift-ingress/payload-processing-e2e-api-b-2ea423-57c64b48d-hb24b"
Aug 02 10:20:06.661141 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:06.660716    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbjt8\" (UniqueName: \"kubernetes.io/projected/13837f5d-7f7d-402a-9d49-9bc8370cf97b-kube-api-access-pbjt8\") pod \"payload-processing-e2e-api-b-2ea423-57c64b48d-hb24b\" (UID: \"13837f5d-7f7d-402a-9d49-9bc8370cf97b\") " pod="openshift-ingress/payload-processing-e2e-api-b-2ea423-57c64b48d-hb24b"
Aug 02 10:20:06.661141 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:06.660755    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d15fe25b-5cd2-4039-8b22-916c89379b75-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2ea423-7fb47f76-thdqn\" (UID: \"d15fe25b-5cd2-4039-8b22-916c89379b75\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ea423-7fb47f76-thdqn"
Aug 02 10:20:06.711026 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:06.711003    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2ea423-9d7c759d4-tbp7x"]
Aug 02 10:20:06.713593 ip-10-0-141-232 kubenswrapper[2564]: W0802 10:20:06.713569    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea557ed5_fdec_41bb_9122_ed2529c8890f.slice/crio-fbff3c1202070a3ac4ed94c8994e76129256182a26e8f0d6c3b84d61176d77ff WatchSource:0}: Error finding container fbff3c1202070a3ac4ed94c8994e76129256182a26e8f0d6c3b84d61176d77ff: Status 404 returned error can't find the container with id fbff3c1202070a3ac4ed94c8994e76129256182a26e8f0d6c3b84d61176d77ff
Aug 02 10:20:06.762045 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:06.762020    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbjt8\" (UniqueName: \"kubernetes.io/projected/13837f5d-7f7d-402a-9d49-9bc8370cf97b-kube-api-access-pbjt8\") pod \"payload-processing-e2e-api-b-2ea423-57c64b48d-hb24b\" (UID: \"13837f5d-7f7d-402a-9d49-9bc8370cf97b\") " pod="openshift-ingress/payload-processing-e2e-api-b-2ea423-57c64b48d-hb24b"
Aug 02 10:20:06.762138 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:06.762057    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d15fe25b-5cd2-4039-8b22-916c89379b75-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2ea423-7fb47f76-thdqn\" (UID: \"d15fe25b-5cd2-4039-8b22-916c89379b75\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ea423-7fb47f76-thdqn"
Aug 02 10:20:06.762138 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:06.762099    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpc5l\" (UniqueName: \"kubernetes.io/projected/d15fe25b-5cd2-4039-8b22-916c89379b75-kube-api-access-jpc5l\") pod \"payload-pre-processing-e2e-api-b-2ea423-7fb47f76-thdqn\" (UID: \"d15fe25b-5cd2-4039-8b22-916c89379b75\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ea423-7fb47f76-thdqn"
Aug 02 10:20:06.762138 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:06.762131    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13837f5d-7f7d-402a-9d49-9bc8370cf97b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2ea423-57c64b48d-hb24b\" (UID: \"13837f5d-7f7d-402a-9d49-9bc8370cf97b\") " pod="openshift-ingress/payload-processing-e2e-api-b-2ea423-57c64b48d-hb24b"
Aug 02 10:20:06.762847 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:06.762831    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d15fe25b-5cd2-4039-8b22-916c89379b75-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2ea423-7fb47f76-thdqn\" (UID: \"d15fe25b-5cd2-4039-8b22-916c89379b75\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ea423-7fb47f76-thdqn"
Aug 02 10:20:06.762895 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:06.762847    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13837f5d-7f7d-402a-9d49-9bc8370cf97b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2ea423-57c64b48d-hb24b\" (UID: \"13837f5d-7f7d-402a-9d49-9bc8370cf97b\") " pod="openshift-ingress/payload-processing-e2e-api-b-2ea423-57c64b48d-hb24b"
Aug 02 10:20:06.769890 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:06.769870    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpc5l\" (UniqueName: \"kubernetes.io/projected/d15fe25b-5cd2-4039-8b22-916c89379b75-kube-api-access-jpc5l\") pod \"payload-pre-processing-e2e-api-b-2ea423-7fb47f76-thdqn\" (UID: \"d15fe25b-5cd2-4039-8b22-916c89379b75\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ea423-7fb47f76-thdqn"
Aug 02 10:20:06.769986 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:06.769912    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbjt8\" (UniqueName: \"kubernetes.io/projected/13837f5d-7f7d-402a-9d49-9bc8370cf97b-kube-api-access-pbjt8\") pod \"payload-processing-e2e-api-b-2ea423-57c64b48d-hb24b\" (UID: \"13837f5d-7f7d-402a-9d49-9bc8370cf97b\") " pod="openshift-ingress/payload-processing-e2e-api-b-2ea423-57c64b48d-hb24b"
Aug 02 10:20:06.841750 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:06.841721    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ea423-7fb47f76-thdqn"
Aug 02 10:20:06.865703 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:06.865684    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2ea423-57c64b48d-hb24b"
Aug 02 10:20:06.986272 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:06.986248    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2ea423-7fb47f76-thdqn"]
Aug 02 10:20:06.988022 ip-10-0-141-232 kubenswrapper[2564]: W0802 10:20:06.987993    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd15fe25b_5cd2_4039_8b22_916c89379b75.slice/crio-045c3680b94fc18574bc7b813aea1117e97b5625ace324d7eaed4d9c634a16e8 WatchSource:0}: Error finding container 045c3680b94fc18574bc7b813aea1117e97b5625ace324d7eaed4d9c634a16e8: Status 404 returned error can't find the container with id 045c3680b94fc18574bc7b813aea1117e97b5625ace324d7eaed4d9c634a16e8
Aug 02 10:20:06.999503 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:06.999478    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ea423-7fb47f76-thdqn" event={"ID":"d15fe25b-5cd2-4039-8b22-916c89379b75","Type":"ContainerStarted","Data":"045c3680b94fc18574bc7b813aea1117e97b5625ace324d7eaed4d9c634a16e8"}
Aug 02 10:20:07.000589 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:07.000566    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ea423-9d7c759d4-tbp7x" event={"ID":"ea557ed5-fdec-41bb-9122-ed2529c8890f","Type":"ContainerStarted","Data":"fbff3c1202070a3ac4ed94c8994e76129256182a26e8f0d6c3b84d61176d77ff"}
Aug 02 10:20:07.012584 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:07.012563    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2ea423-57c64b48d-hb24b"]
Aug 02 10:20:07.024184 ip-10-0-141-232 kubenswrapper[2564]: W0802 10:20:07.024156    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13837f5d_7f7d_402a_9d49_9bc8370cf97b.slice/crio-8b36201b8ce4b0caafb6df679c1d11c22d0124c5625743caf58917c4816893c7 WatchSource:0}: Error finding container 8b36201b8ce4b0caafb6df679c1d11c22d0124c5625743caf58917c4816893c7: Status 404 returned error can't find the container with id 8b36201b8ce4b0caafb6df679c1d11c22d0124c5625743caf58917c4816893c7
Aug 02 10:20:08.005445 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:08.005409    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ea423-7fb47f76-thdqn" event={"ID":"d15fe25b-5cd2-4039-8b22-916c89379b75","Type":"ContainerStarted","Data":"146a496268c924d22c5308a4efcc8f25a12e1b2825e5834b4667d5a7a29f5fc0"}
Aug 02 10:20:08.005877 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:08.005490    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ea423-7fb47f76-thdqn"
Aug 02 10:20:08.006852 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:08.006819    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ea423-9d7c759d4-tbp7x" event={"ID":"ea557ed5-fdec-41bb-9122-ed2529c8890f","Type":"ContainerStarted","Data":"72f5c3bc0eaa78850c860ac43934c1ec09e9bfa94b2bdd1f018c1357da6feef1"}
Aug 02 10:20:08.006977 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:08.006873    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ea423-9d7c759d4-tbp7x"
Aug 02 10:20:08.008095 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:08.008073    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2ea423-57c64b48d-hb24b" event={"ID":"13837f5d-7f7d-402a-9d49-9bc8370cf97b","Type":"ContainerStarted","Data":"f33ad54fe408bdd3312785c47c69d5e81156578d3324649d1f23722b64d3ea52"}
Aug 02 10:20:08.008204 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:08.008110    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2ea423-57c64b48d-hb24b" event={"ID":"13837f5d-7f7d-402a-9d49-9bc8370cf97b","Type":"ContainerStarted","Data":"8b36201b8ce4b0caafb6df679c1d11c22d0124c5625743caf58917c4816893c7"}
Aug 02 10:20:08.008204 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:08.008195    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-2ea423-57c64b48d-hb24b"
Aug 02 10:20:08.024596 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:08.024560    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ea423-7fb47f76-thdqn" podStartSLOduration=2.024548179 podStartE2EDuration="2.024548179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:20:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:20:08.023297031 +0000 UTC m=+2831.738102480" watchObservedRunningTime="2026-08-02 10:20:08.024548179 +0000 UTC m=+2831.739353686"
Aug 02 10:20:08.044991 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:08.044947    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ea423-9d7c759d4-tbp7x" podStartSLOduration=2.717892509 podStartE2EDuration="3.044936102s" totalImagesPullingTime="327.043593ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:20:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:20:08.042883384 +0000 UTC m=+2831.757688832" watchObservedRunningTime="2026-08-02 10:20:08.044936102 +0000 UTC m=+2831.759741550"
Aug 02 10:20:08.065405 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:08.065365    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-2ea423-57c64b48d-hb24b" podStartSLOduration=2.065354323 podStartE2EDuration="2.065354323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:20:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:20:08.065039256 +0000 UTC m=+2831.779844704" watchObservedRunningTime="2026-08-02 10:20:08.065354323 +0000 UTC m=+2831.780159770"
Aug 02 10:20:14.019012 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:14.018983    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ea423-9d7c759d4-tbp7x"
Aug 02 10:20:19.014516 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:19.014477    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ea423-7fb47f76-thdqn"
Aug 02 10:20:19.014902 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:19.014582    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-2ea423-57c64b48d-hb24b"
Aug 02 10:20:29.532839 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:29.532797    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:20:47.966419 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:47.966364    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-931e8a-68c4d5c44c-8vj5v"]
Aug 02 10:20:47.967011 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:47.966649    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-931e8a-68c4d5c44c-8vj5v" podUID="1877b800-7c5f-4298-8eb2-b45189118c2a" containerName="maas-api" containerID="cri-o://56ba67c9179fcd6cca1c2c45565822c51336f038d52f5ef325aa3def591b3bfc" gracePeriod=30
Aug 02 10:20:47.995650 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:47.995618    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:20:48.017688 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:48.017660    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-931e8a-6fb9976b98-bc8hp"]
Aug 02 10:20:48.017908 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:48.017859    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-931e8a-6fb9976b98-bc8hp" podUID="c8479024-c2a6-4b33-b403-472e80654ebe" containerName="payload-processing" containerID="cri-o://01db110f68d7d6c9e4ef16fd1886d7cfd10a9243667ecba5c3c916b07f66d992" gracePeriod=30
Aug 02 10:20:48.029645 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:48.029621    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-931e8a-6986bf6c84-d5zsn"]
Aug 02 10:20:48.029850 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:48.029829    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-931e8a-6986bf6c84-d5zsn" podUID="3690f42d-b0f6-4558-a7de-a5cfa3df7cb8" containerName="payload-pre-processing" containerID="cri-o://8eb031958454b6269c57632947e66b204694756816a10246e823e4ee784d55be" gracePeriod=30
Aug 02 10:20:48.096277 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:48.096227    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:20:48.176673 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:48.176634    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:20:48.180978 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:48.180861    2564 generic.go:356] "Generic (PLEG): container finished" podID="1877b800-7c5f-4298-8eb2-b45189118c2a" containerID="56ba67c9179fcd6cca1c2c45565822c51336f038d52f5ef325aa3def591b3bfc" exitCode=0
Aug 02 10:20:48.180978 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:48.180929    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-931e8a-68c4d5c44c-8vj5v" event={"ID":"1877b800-7c5f-4298-8eb2-b45189118c2a","Type":"ContainerDied","Data":"56ba67c9179fcd6cca1c2c45565822c51336f038d52f5ef325aa3def591b3bfc"}
Aug 02 10:20:48.182629 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:48.182601    2564 generic.go:356] "Generic (PLEG): container finished" podID="c8479024-c2a6-4b33-b403-472e80654ebe" containerID="01db110f68d7d6c9e4ef16fd1886d7cfd10a9243667ecba5c3c916b07f66d992" exitCode=0
Aug 02 10:20:48.182772 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:48.182675    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-931e8a-6fb9976b98-bc8hp" event={"ID":"c8479024-c2a6-4b33-b403-472e80654ebe","Type":"ContainerDied","Data":"01db110f68d7d6c9e4ef16fd1886d7cfd10a9243667ecba5c3c916b07f66d992"}
Aug 02 10:20:48.184426 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:48.184392    2564 generic.go:356] "Generic (PLEG): container finished" podID="3690f42d-b0f6-4558-a7de-a5cfa3df7cb8" containerID="8eb031958454b6269c57632947e66b204694756816a10246e823e4ee784d55be" exitCode=0
Aug 02 10:20:48.184544 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:48.184430    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-931e8a-6986bf6c84-d5zsn" event={"ID":"3690f42d-b0f6-4558-a7de-a5cfa3df7cb8","Type":"ContainerDied","Data":"8eb031958454b6269c57632947e66b204694756816a10246e823e4ee784d55be"}
Aug 02 10:20:48.239138 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:48.239057    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:20:48.314701 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:48.314672    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-931e8a-6fb9976b98-bc8hp"
Aug 02 10:20:48.317346 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:48.317327    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-931e8a-6986bf6c84-d5zsn"
Aug 02 10:20:48.320476 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:48.320458    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-931e8a-68c4d5c44c-8vj5v"
Aug 02 10:20:48.390758 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:48.390727    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6qwn2\" (UniqueName: \"kubernetes.io/projected/c8479024-c2a6-4b33-b403-472e80654ebe-kube-api-access-6qwn2\") pod \"c8479024-c2a6-4b33-b403-472e80654ebe\" (UID: \"c8479024-c2a6-4b33-b403-472e80654ebe\") "
Aug 02 10:20:48.390912 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:48.390764    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7p7jb\" (UniqueName: \"kubernetes.io/projected/1877b800-7c5f-4298-8eb2-b45189118c2a-kube-api-access-7p7jb\") pod \"1877b800-7c5f-4298-8eb2-b45189118c2a\" (UID: \"1877b800-7c5f-4298-8eb2-b45189118c2a\") "
Aug 02 10:20:48.390912 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:48.390784    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8479024-c2a6-4b33-b403-472e80654ebe-plugins-config-volume\") pod \"c8479024-c2a6-4b33-b403-472e80654ebe\" (UID: \"c8479024-c2a6-4b33-b403-472e80654ebe\") "
Aug 02 10:20:48.390912 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:48.390804    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1877b800-7c5f-4298-8eb2-b45189118c2a-maas-api-tls\") pod \"1877b800-7c5f-4298-8eb2-b45189118c2a\" (UID: \"1877b800-7c5f-4298-8eb2-b45189118c2a\") "
Aug 02 10:20:48.390912 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:48.390831    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1877b800-7c5f-4298-8eb2-b45189118c2a-openshift-service-ca\") pod \"1877b800-7c5f-4298-8eb2-b45189118c2a\" (UID: \"1877b800-7c5f-4298-8eb2-b45189118c2a\") "
Aug 02 10:20:48.390912 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:48.390882    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3690f42d-b0f6-4558-a7de-a5cfa3df7cb8-plugins-config-volume\") pod \"3690f42d-b0f6-4558-a7de-a5cfa3df7cb8\" (UID: \"3690f42d-b0f6-4558-a7de-a5cfa3df7cb8\") "
Aug 02 10:20:48.391214 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:48.390943    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mq6t7\" (UniqueName: \"kubernetes.io/projected/3690f42d-b0f6-4558-a7de-a5cfa3df7cb8-kube-api-access-mq6t7\") pod \"3690f42d-b0f6-4558-a7de-a5cfa3df7cb8\" (UID: \"3690f42d-b0f6-4558-a7de-a5cfa3df7cb8\") "
Aug 02 10:20:48.391292 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:48.391204    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8479024-c2a6-4b33-b403-472e80654ebe-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c8479024-c2a6-4b33-b403-472e80654ebe" (UID: "c8479024-c2a6-4b33-b403-472e80654ebe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 02 10:20:48.391292 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:48.391256    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1877b800-7c5f-4298-8eb2-b45189118c2a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1877b800-7c5f-4298-8eb2-b45189118c2a" (UID: "1877b800-7c5f-4298-8eb2-b45189118c2a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 02 10:20:48.391292 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:48.391272    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3690f42d-b0f6-4558-a7de-a5cfa3df7cb8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3690f42d-b0f6-4558-a7de-a5cfa3df7cb8" (UID: "3690f42d-b0f6-4558-a7de-a5cfa3df7cb8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 02 10:20:48.391527 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:48.391491    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8479024-c2a6-4b33-b403-472e80654ebe-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:20:48.391527 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:48.391514    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1877b800-7c5f-4298-8eb2-b45189118c2a-openshift-service-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:20:48.391647 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:48.391529    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3690f42d-b0f6-4558-a7de-a5cfa3df7cb8-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:20:48.392904 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:48.392873    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1877b800-7c5f-4298-8eb2-b45189118c2a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1877b800-7c5f-4298-8eb2-b45189118c2a" (UID: "1877b800-7c5f-4298-8eb2-b45189118c2a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 02 10:20:48.393008 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:48.392942    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8479024-c2a6-4b33-b403-472e80654ebe-kube-api-access-6qwn2" (OuterVolumeSpecName: "kube-api-access-6qwn2") pod "c8479024-c2a6-4b33-b403-472e80654ebe" (UID: "c8479024-c2a6-4b33-b403-472e80654ebe"). InnerVolumeSpecName "kube-api-access-6qwn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 02 10:20:48.393008 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:48.392961    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3690f42d-b0f6-4558-a7de-a5cfa3df7cb8-kube-api-access-mq6t7" (OuterVolumeSpecName: "kube-api-access-mq6t7") pod "3690f42d-b0f6-4558-a7de-a5cfa3df7cb8" (UID: "3690f42d-b0f6-4558-a7de-a5cfa3df7cb8"). InnerVolumeSpecName "kube-api-access-mq6t7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 02 10:20:48.393008 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:48.392989    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1877b800-7c5f-4298-8eb2-b45189118c2a-kube-api-access-7p7jb" (OuterVolumeSpecName: "kube-api-access-7p7jb") pod "1877b800-7c5f-4298-8eb2-b45189118c2a" (UID: "1877b800-7c5f-4298-8eb2-b45189118c2a"). InnerVolumeSpecName "kube-api-access-7p7jb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 02 10:20:48.492318 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:48.492256    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq6t7\" (UniqueName: \"kubernetes.io/projected/3690f42d-b0f6-4558-a7de-a5cfa3df7cb8-kube-api-access-mq6t7\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:20:48.492318 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:48.492280    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qwn2\" (UniqueName: \"kubernetes.io/projected/c8479024-c2a6-4b33-b403-472e80654ebe-kube-api-access-6qwn2\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:20:48.492318 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:48.492289    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p7jb\" (UniqueName: \"kubernetes.io/projected/1877b800-7c5f-4298-8eb2-b45189118c2a-kube-api-access-7p7jb\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:20:48.492318 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:48.492298    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1877b800-7c5f-4298-8eb2-b45189118c2a-maas-api-tls\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:20:48.555265 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:48.555220    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:20:49.189789 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:49.189675    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-931e8a-6986bf6c84-d5zsn"
Aug 02 10:20:49.189789 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:49.189686    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-931e8a-6986bf6c84-d5zsn" event={"ID":"3690f42d-b0f6-4558-a7de-a5cfa3df7cb8","Type":"ContainerDied","Data":"d5c5d8c99f8fe43bcda2016349e76d3210b6273f73df08a4cdd35d433776708f"}
Aug 02 10:20:49.189789 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:49.189786    2564 scope.go:117] "RemoveContainer" containerID="8eb031958454b6269c57632947e66b204694756816a10246e823e4ee784d55be"
Aug 02 10:20:49.191172 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:49.191149    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-931e8a-68c4d5c44c-8vj5v"
Aug 02 10:20:49.191314 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:49.191176    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-931e8a-68c4d5c44c-8vj5v" event={"ID":"1877b800-7c5f-4298-8eb2-b45189118c2a","Type":"ContainerDied","Data":"40d59d86a30dfbe22a297e79b67bfd75c575a688acb052bc7987d95cdb408bb9"}
Aug 02 10:20:49.192625 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:49.192601    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-931e8a-6fb9976b98-bc8hp" event={"ID":"c8479024-c2a6-4b33-b403-472e80654ebe","Type":"ContainerDied","Data":"2979e4e35f0f035db9851d2f779bb56faf14a322db28aeeb60879932ed92e1ff"}
Aug 02 10:20:49.192753 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:49.192644    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-931e8a-6fb9976b98-bc8hp"
Aug 02 10:20:49.199195 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:49.199176    2564 scope.go:117] "RemoveContainer" containerID="56ba67c9179fcd6cca1c2c45565822c51336f038d52f5ef325aa3def591b3bfc"
Aug 02 10:20:49.206576 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:49.206555    2564 scope.go:117] "RemoveContainer" containerID="01db110f68d7d6c9e4ef16fd1886d7cfd10a9243667ecba5c3c916b07f66d992"
Aug 02 10:20:49.216734 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:49.216710    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-931e8a-6986bf6c84-d5zsn"]
Aug 02 10:20:49.218727 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:49.218707    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-931e8a-6986bf6c84-d5zsn"]
Aug 02 10:20:49.230159 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:49.230137    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-931e8a-6fb9976b98-bc8hp"]
Aug 02 10:20:49.234363 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:49.234342    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-931e8a-6fb9976b98-bc8hp"]
Aug 02 10:20:49.245488 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:49.245470    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-931e8a-68c4d5c44c-8vj5v"]
Aug 02 10:20:49.250029 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:49.250008    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-931e8a-68c4d5c44c-8vj5v"]
Aug 02 10:20:50.880630 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:50.880600    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1877b800-7c5f-4298-8eb2-b45189118c2a" path="/var/lib/kubelet/pods/1877b800-7c5f-4298-8eb2-b45189118c2a/volumes"
Aug 02 10:20:50.880996 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:50.880936    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3690f42d-b0f6-4558-a7de-a5cfa3df7cb8" path="/var/lib/kubelet/pods/3690f42d-b0f6-4558-a7de-a5cfa3df7cb8/volumes"
Aug 02 10:20:50.881246 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:50.881223    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8479024-c2a6-4b33-b403-472e80654ebe" path="/var/lib/kubelet/pods/c8479024-c2a6-4b33-b403-472e80654ebe/volumes"
Aug 02 10:20:54.419184 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:54.419147    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:20:54.519810 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:54.519779    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:20:54.684802 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:54.684731    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:20:54.916681 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:20:54.916646    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:21:10.469705 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:10.469669    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2ea423-9d7c759d4-tbp7x"]
Aug 02 10:21:10.470143 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:10.469965    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ea423-9d7c759d4-tbp7x" podUID="ea557ed5-fdec-41bb-9122-ed2529c8890f" containerName="maas-api" containerID="cri-o://72f5c3bc0eaa78850c860ac43934c1ec09e9bfa94b2bdd1f018c1357da6feef1" gracePeriod=30
Aug 02 10:21:10.504241 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:10.504208    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:21:10.528298 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:10.528271    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2ea423-57c64b48d-hb24b"]
Aug 02 10:21:10.528548 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:10.528516    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-2ea423-57c64b48d-hb24b" podUID="13837f5d-7f7d-402a-9d49-9bc8370cf97b" containerName="payload-processing" containerID="cri-o://f33ad54fe408bdd3312785c47c69d5e81156578d3324649d1f23722b64d3ea52" gracePeriod=30
Aug 02 10:21:10.533566 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:10.533540    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2ea423-7fb47f76-thdqn"]
Aug 02 10:21:10.533781 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:10.533757    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ea423-7fb47f76-thdqn" podUID="d15fe25b-5cd2-4039-8b22-916c89379b75" containerName="payload-pre-processing" containerID="cri-o://146a496268c924d22c5308a4efcc8f25a12e1b2825e5834b4667d5a7a29f5fc0" gracePeriod=30
Aug 02 10:21:10.605176 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:10.605144    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:21:10.683653 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:10.683620    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:21:10.744480 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:10.744451    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:21:10.823388 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:10.823367    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ea423-9d7c759d4-tbp7x"
Aug 02 10:21:10.826377 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:10.826355    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2ea423-57c64b48d-hb24b"
Aug 02 10:21:10.836965 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:10.836946    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ea423-7fb47f76-thdqn"
Aug 02 10:21:10.864601 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:10.864567    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbjt8\" (UniqueName: \"kubernetes.io/projected/13837f5d-7f7d-402a-9d49-9bc8370cf97b-kube-api-access-pbjt8\") pod \"13837f5d-7f7d-402a-9d49-9bc8370cf97b\" (UID: \"13837f5d-7f7d-402a-9d49-9bc8370cf97b\") "
Aug 02 10:21:10.864733 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:10.864623    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13837f5d-7f7d-402a-9d49-9bc8370cf97b-plugins-config-volume\") pod \"13837f5d-7f7d-402a-9d49-9bc8370cf97b\" (UID: \"13837f5d-7f7d-402a-9d49-9bc8370cf97b\") "
Aug 02 10:21:10.864733 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:10.864660    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kb5kj\" (UniqueName: \"kubernetes.io/projected/ea557ed5-fdec-41bb-9122-ed2529c8890f-kube-api-access-kb5kj\") pod \"ea557ed5-fdec-41bb-9122-ed2529c8890f\" (UID: \"ea557ed5-fdec-41bb-9122-ed2529c8890f\") "
Aug 02 10:21:10.864733 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:10.864699    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea557ed5-fdec-41bb-9122-ed2529c8890f-maas-api-tls\") pod \"ea557ed5-fdec-41bb-9122-ed2529c8890f\" (UID: \"ea557ed5-fdec-41bb-9122-ed2529c8890f\") "
Aug 02 10:21:10.864733 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:10.864731    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea557ed5-fdec-41bb-9122-ed2529c8890f-openshift-service-ca\") pod \"ea557ed5-fdec-41bb-9122-ed2529c8890f\" (UID: \"ea557ed5-fdec-41bb-9122-ed2529c8890f\") "
Aug 02 10:21:10.865033 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:10.865012    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13837f5d-7f7d-402a-9d49-9bc8370cf97b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "13837f5d-7f7d-402a-9d49-9bc8370cf97b" (UID: "13837f5d-7f7d-402a-9d49-9bc8370cf97b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 02 10:21:10.865221 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:10.865199    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea557ed5-fdec-41bb-9122-ed2529c8890f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ea557ed5-fdec-41bb-9122-ed2529c8890f" (UID: "ea557ed5-fdec-41bb-9122-ed2529c8890f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 02 10:21:10.866681 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:10.866660    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13837f5d-7f7d-402a-9d49-9bc8370cf97b-kube-api-access-pbjt8" (OuterVolumeSpecName: "kube-api-access-pbjt8") pod "13837f5d-7f7d-402a-9d49-9bc8370cf97b" (UID: "13837f5d-7f7d-402a-9d49-9bc8370cf97b"). InnerVolumeSpecName "kube-api-access-pbjt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 02 10:21:10.867185 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:10.867160    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea557ed5-fdec-41bb-9122-ed2529c8890f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ea557ed5-fdec-41bb-9122-ed2529c8890f" (UID: "ea557ed5-fdec-41bb-9122-ed2529c8890f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 02 10:21:10.867339 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:10.867160    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea557ed5-fdec-41bb-9122-ed2529c8890f-kube-api-access-kb5kj" (OuterVolumeSpecName: "kube-api-access-kb5kj") pod "ea557ed5-fdec-41bb-9122-ed2529c8890f" (UID: "ea557ed5-fdec-41bb-9122-ed2529c8890f"). InnerVolumeSpecName "kube-api-access-kb5kj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 02 10:21:10.965403 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:10.965373    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpc5l\" (UniqueName: \"kubernetes.io/projected/d15fe25b-5cd2-4039-8b22-916c89379b75-kube-api-access-jpc5l\") pod \"d15fe25b-5cd2-4039-8b22-916c89379b75\" (UID: \"d15fe25b-5cd2-4039-8b22-916c89379b75\") "
Aug 02 10:21:10.965534 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:10.965410    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d15fe25b-5cd2-4039-8b22-916c89379b75-plugins-config-volume\") pod \"d15fe25b-5cd2-4039-8b22-916c89379b75\" (UID: \"d15fe25b-5cd2-4039-8b22-916c89379b75\") "
Aug 02 10:21:10.965634 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:10.965619    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbjt8\" (UniqueName: \"kubernetes.io/projected/13837f5d-7f7d-402a-9d49-9bc8370cf97b-kube-api-access-pbjt8\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:21:10.965701 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:10.965641    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13837f5d-7f7d-402a-9d49-9bc8370cf97b-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:21:10.965701 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:10.965656    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb5kj\" (UniqueName: \"kubernetes.io/projected/ea557ed5-fdec-41bb-9122-ed2529c8890f-kube-api-access-kb5kj\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:21:10.965701 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:10.965671    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea557ed5-fdec-41bb-9122-ed2529c8890f-maas-api-tls\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:21:10.965701 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:10.965684    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea557ed5-fdec-41bb-9122-ed2529c8890f-openshift-service-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:21:10.965834 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:10.965754    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d15fe25b-5cd2-4039-8b22-916c89379b75-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d15fe25b-5cd2-4039-8b22-916c89379b75" (UID: "d15fe25b-5cd2-4039-8b22-916c89379b75"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 02 10:21:10.967373 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:10.967345    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d15fe25b-5cd2-4039-8b22-916c89379b75-kube-api-access-jpc5l" (OuterVolumeSpecName: "kube-api-access-jpc5l") pod "d15fe25b-5cd2-4039-8b22-916c89379b75" (UID: "d15fe25b-5cd2-4039-8b22-916c89379b75"). InnerVolumeSpecName "kube-api-access-jpc5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 02 10:21:11.059968 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:11.059903    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:21:11.066879 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:11.066861    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpc5l\" (UniqueName: \"kubernetes.io/projected/d15fe25b-5cd2-4039-8b22-916c89379b75-kube-api-access-jpc5l\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:21:11.066949 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:11.066882    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d15fe25b-5cd2-4039-8b22-916c89379b75-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:21:11.287978 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:11.287945    2564 generic.go:356] "Generic (PLEG): container finished" podID="13837f5d-7f7d-402a-9d49-9bc8370cf97b" containerID="f33ad54fe408bdd3312785c47c69d5e81156578d3324649d1f23722b64d3ea52" exitCode=0
Aug 02 10:21:11.288135 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:11.288001    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2ea423-57c64b48d-hb24b"
Aug 02 10:21:11.288135 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:11.288027    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2ea423-57c64b48d-hb24b" event={"ID":"13837f5d-7f7d-402a-9d49-9bc8370cf97b","Type":"ContainerDied","Data":"f33ad54fe408bdd3312785c47c69d5e81156578d3324649d1f23722b64d3ea52"}
Aug 02 10:21:11.288135 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:11.288081    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2ea423-57c64b48d-hb24b" event={"ID":"13837f5d-7f7d-402a-9d49-9bc8370cf97b","Type":"ContainerDied","Data":"8b36201b8ce4b0caafb6df679c1d11c22d0124c5625743caf58917c4816893c7"}
Aug 02 10:21:11.288135 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:11.288112    2564 scope.go:117] "RemoveContainer" containerID="f33ad54fe408bdd3312785c47c69d5e81156578d3324649d1f23722b64d3ea52"
Aug 02 10:21:11.289286 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:11.289263    2564 generic.go:356] "Generic (PLEG): container finished" podID="d15fe25b-5cd2-4039-8b22-916c89379b75" containerID="146a496268c924d22c5308a4efcc8f25a12e1b2825e5834b4667d5a7a29f5fc0" exitCode=0
Aug 02 10:21:11.289414 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:11.289333    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ea423-7fb47f76-thdqn"
Aug 02 10:21:11.289414 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:11.289344    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ea423-7fb47f76-thdqn" event={"ID":"d15fe25b-5cd2-4039-8b22-916c89379b75","Type":"ContainerDied","Data":"146a496268c924d22c5308a4efcc8f25a12e1b2825e5834b4667d5a7a29f5fc0"}
Aug 02 10:21:11.289414 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:11.289398    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ea423-7fb47f76-thdqn" event={"ID":"d15fe25b-5cd2-4039-8b22-916c89379b75","Type":"ContainerDied","Data":"045c3680b94fc18574bc7b813aea1117e97b5625ace324d7eaed4d9c634a16e8"}
Aug 02 10:21:11.290867 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:11.290842    2564 generic.go:356] "Generic (PLEG): container finished" podID="ea557ed5-fdec-41bb-9122-ed2529c8890f" containerID="72f5c3bc0eaa78850c860ac43934c1ec09e9bfa94b2bdd1f018c1357da6feef1" exitCode=0
Aug 02 10:21:11.290982 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:11.290890    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ea423-9d7c759d4-tbp7x"
Aug 02 10:21:11.290982 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:11.290914    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ea423-9d7c759d4-tbp7x" event={"ID":"ea557ed5-fdec-41bb-9122-ed2529c8890f","Type":"ContainerDied","Data":"72f5c3bc0eaa78850c860ac43934c1ec09e9bfa94b2bdd1f018c1357da6feef1"}
Aug 02 10:21:11.290982 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:11.290956    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ea423-9d7c759d4-tbp7x" event={"ID":"ea557ed5-fdec-41bb-9122-ed2529c8890f","Type":"ContainerDied","Data":"fbff3c1202070a3ac4ed94c8994e76129256182a26e8f0d6c3b84d61176d77ff"}
Aug 02 10:21:11.296357 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:11.296341    2564 scope.go:117] "RemoveContainer" containerID="f33ad54fe408bdd3312785c47c69d5e81156578d3324649d1f23722b64d3ea52"
Aug 02 10:21:11.296648 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:21:11.296627    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f33ad54fe408bdd3312785c47c69d5e81156578d3324649d1f23722b64d3ea52\": container with ID starting with f33ad54fe408bdd3312785c47c69d5e81156578d3324649d1f23722b64d3ea52 not found: ID does not exist" containerID="f33ad54fe408bdd3312785c47c69d5e81156578d3324649d1f23722b64d3ea52"
Aug 02 10:21:11.296709 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:11.296658    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f33ad54fe408bdd3312785c47c69d5e81156578d3324649d1f23722b64d3ea52"} err="failed to get container status \"f33ad54fe408bdd3312785c47c69d5e81156578d3324649d1f23722b64d3ea52\": rpc error: code = NotFound desc = could not find container \"f33ad54fe408bdd3312785c47c69d5e81156578d3324649d1f23722b64d3ea52\": container with ID starting with f33ad54fe408bdd3312785c47c69d5e81156578d3324649d1f23722b64d3ea52 not found: ID does not exist"
Aug 02 10:21:11.296709 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:11.296680    2564 scope.go:117] "RemoveContainer" containerID="146a496268c924d22c5308a4efcc8f25a12e1b2825e5834b4667d5a7a29f5fc0"
Aug 02 10:21:11.303972 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:11.303953    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2ea423-57c64b48d-hb24b"]
Aug 02 10:21:11.304533 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:11.304519    2564 scope.go:117] "RemoveContainer" containerID="146a496268c924d22c5308a4efcc8f25a12e1b2825e5834b4667d5a7a29f5fc0"
Aug 02 10:21:11.304765 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:21:11.304740    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"146a496268c924d22c5308a4efcc8f25a12e1b2825e5834b4667d5a7a29f5fc0\": container with ID starting with 146a496268c924d22c5308a4efcc8f25a12e1b2825e5834b4667d5a7a29f5fc0 not found: ID does not exist" containerID="146a496268c924d22c5308a4efcc8f25a12e1b2825e5834b4667d5a7a29f5fc0"
Aug 02 10:21:11.304856 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:11.304768    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"146a496268c924d22c5308a4efcc8f25a12e1b2825e5834b4667d5a7a29f5fc0"} err="failed to get container status \"146a496268c924d22c5308a4efcc8f25a12e1b2825e5834b4667d5a7a29f5fc0\": rpc error: code = NotFound desc = could not find container \"146a496268c924d22c5308a4efcc8f25a12e1b2825e5834b4667d5a7a29f5fc0\": container with ID starting with 146a496268c924d22c5308a4efcc8f25a12e1b2825e5834b4667d5a7a29f5fc0 not found: ID does not exist"
Aug 02 10:21:11.304856 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:11.304784    2564 scope.go:117] "RemoveContainer" containerID="72f5c3bc0eaa78850c860ac43934c1ec09e9bfa94b2bdd1f018c1357da6feef1"
Aug 02 10:21:11.308298 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:11.308278    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2ea423-57c64b48d-hb24b"]
Aug 02 10:21:11.311869 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:11.311853    2564 scope.go:117] "RemoveContainer" containerID="72f5c3bc0eaa78850c860ac43934c1ec09e9bfa94b2bdd1f018c1357da6feef1"
Aug 02 10:21:11.312150 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:21:11.312130    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72f5c3bc0eaa78850c860ac43934c1ec09e9bfa94b2bdd1f018c1357da6feef1\": container with ID starting with 72f5c3bc0eaa78850c860ac43934c1ec09e9bfa94b2bdd1f018c1357da6feef1 not found: ID does not exist" containerID="72f5c3bc0eaa78850c860ac43934c1ec09e9bfa94b2bdd1f018c1357da6feef1"
Aug 02 10:21:11.312204 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:11.312154    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72f5c3bc0eaa78850c860ac43934c1ec09e9bfa94b2bdd1f018c1357da6feef1"} err="failed to get container status \"72f5c3bc0eaa78850c860ac43934c1ec09e9bfa94b2bdd1f018c1357da6feef1\": rpc error: code = NotFound desc = could not find container \"72f5c3bc0eaa78850c860ac43934c1ec09e9bfa94b2bdd1f018c1357da6feef1\": container with ID starting with 72f5c3bc0eaa78850c860ac43934c1ec09e9bfa94b2bdd1f018c1357da6feef1 not found: ID does not exist"
Aug 02 10:21:11.324009 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:11.323987    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2ea423-7fb47f76-thdqn"]
Aug 02 10:21:11.329221 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:11.329203    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2ea423-7fb47f76-thdqn"]
Aug 02 10:21:11.340197 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:11.340175    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2ea423-9d7c759d4-tbp7x"]
Aug 02 10:21:11.344223 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:11.344205    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2ea423-9d7c759d4-tbp7x"]
Aug 02 10:21:12.880449 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:12.880413    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13837f5d-7f7d-402a-9d49-9bc8370cf97b" path="/var/lib/kubelet/pods/13837f5d-7f7d-402a-9d49-9bc8370cf97b/volumes"
Aug 02 10:21:12.880942 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:12.880877    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d15fe25b-5cd2-4039-8b22-916c89379b75" path="/var/lib/kubelet/pods/d15fe25b-5cd2-4039-8b22-916c89379b75/volumes"
Aug 02 10:21:12.881289 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:12.881271    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea557ed5-fdec-41bb-9122-ed2529c8890f" path="/var/lib/kubelet/pods/ea557ed5-fdec-41bb-9122-ed2529c8890f/volumes"
Aug 02 10:21:17.078400 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:17.078359    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:21:17.156004 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:17.155968    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:21:17.366364 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:17.366282    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:21:17.601407 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:17.601378    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:21:17.945125 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:17.945089    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:21:18.295686 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:18.295609    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:21:18.603824 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:18.603735    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:21:20.891584 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:20.891540    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:21:20.985856 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:20.985825    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:21:21.984698 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:21.984663    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e9225c-76cf7db755-rc2bb"]
Aug 02 10:21:21.985061 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:21.984948    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8479024-c2a6-4b33-b403-472e80654ebe" containerName="payload-processing"
Aug 02 10:21:21.985061 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:21.984958    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8479024-c2a6-4b33-b403-472e80654ebe" containerName="payload-processing"
Aug 02 10:21:21.985061 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:21.984968    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13837f5d-7f7d-402a-9d49-9bc8370cf97b" containerName="payload-processing"
Aug 02 10:21:21.985061 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:21.984974    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="13837f5d-7f7d-402a-9d49-9bc8370cf97b" containerName="payload-processing"
Aug 02 10:21:21.985061 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:21.984981    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1877b800-7c5f-4298-8eb2-b45189118c2a" containerName="maas-api"
Aug 02 10:21:21.985061 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:21.984987    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1877b800-7c5f-4298-8eb2-b45189118c2a" containerName="maas-api"
Aug 02 10:21:21.985061 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:21.984997    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3690f42d-b0f6-4558-a7de-a5cfa3df7cb8" containerName="payload-pre-processing"
Aug 02 10:21:21.985061 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:21.985002    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3690f42d-b0f6-4558-a7de-a5cfa3df7cb8" containerName="payload-pre-processing"
Aug 02 10:21:21.985061 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:21.985011    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea557ed5-fdec-41bb-9122-ed2529c8890f" containerName="maas-api"
Aug 02 10:21:21.985061 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:21.985016    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea557ed5-fdec-41bb-9122-ed2529c8890f" containerName="maas-api"
Aug 02 10:21:21.985061 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:21.985022    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d15fe25b-5cd2-4039-8b22-916c89379b75" containerName="payload-pre-processing"
Aug 02 10:21:21.985061 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:21.985028    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d15fe25b-5cd2-4039-8b22-916c89379b75" containerName="payload-pre-processing"
Aug 02 10:21:21.985457 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:21.985071    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3690f42d-b0f6-4558-a7de-a5cfa3df7cb8" containerName="payload-pre-processing"
Aug 02 10:21:21.985457 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:21.985079    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea557ed5-fdec-41bb-9122-ed2529c8890f" containerName="maas-api"
Aug 02 10:21:21.985457 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:21.985085    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8479024-c2a6-4b33-b403-472e80654ebe" containerName="payload-processing"
Aug 02 10:21:21.985457 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:21.985093    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1877b800-7c5f-4298-8eb2-b45189118c2a" containerName="maas-api"
Aug 02 10:21:21.985457 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:21.985098    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d15fe25b-5cd2-4039-8b22-916c89379b75" containerName="payload-pre-processing"
Aug 02 10:21:21.985457 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:21.985103    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="13837f5d-7f7d-402a-9d49-9bc8370cf97b" containerName="payload-processing"
Aug 02 10:21:21.989268 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:21.989225    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e9225c-76cf7db755-rc2bb"
Aug 02 10:21:21.991639 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:21.991617    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-e9225c-serving-cert\""
Aug 02 10:21:21.998158 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:21.998135    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e9225c-76cf7db755-rc2bb"]
Aug 02 10:21:22.018200 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:22.018176    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:21:22.047163 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:22.047141    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f970ab0-becb-42e7-bdf0-e04a8267c531-openshift-service-ca\") pod \"maas-api-e2e-shared-a-e9225c-76cf7db755-rc2bb\" (UID: \"0f970ab0-becb-42e7-bdf0-e04a8267c531\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e9225c-76cf7db755-rc2bb"
Aug 02 10:21:22.047292 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:22.047174    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4h8s\" (UniqueName: \"kubernetes.io/projected/0f970ab0-becb-42e7-bdf0-e04a8267c531-kube-api-access-s4h8s\") pod \"maas-api-e2e-shared-a-e9225c-76cf7db755-rc2bb\" (UID: \"0f970ab0-becb-42e7-bdf0-e04a8267c531\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e9225c-76cf7db755-rc2bb"
Aug 02 10:21:22.047292 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:22.047218    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f970ab0-becb-42e7-bdf0-e04a8267c531-maas-api-tls\") pod \"maas-api-e2e-shared-a-e9225c-76cf7db755-rc2bb\" (UID: \"0f970ab0-becb-42e7-bdf0-e04a8267c531\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e9225c-76cf7db755-rc2bb"
Aug 02 10:21:22.148354 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:22.148323    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f970ab0-becb-42e7-bdf0-e04a8267c531-maas-api-tls\") pod \"maas-api-e2e-shared-a-e9225c-76cf7db755-rc2bb\" (UID: \"0f970ab0-becb-42e7-bdf0-e04a8267c531\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e9225c-76cf7db755-rc2bb"
Aug 02 10:21:22.148481 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:22.148406    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f970ab0-becb-42e7-bdf0-e04a8267c531-openshift-service-ca\") pod \"maas-api-e2e-shared-a-e9225c-76cf7db755-rc2bb\" (UID: \"0f970ab0-becb-42e7-bdf0-e04a8267c531\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e9225c-76cf7db755-rc2bb"
Aug 02 10:21:22.148481 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:22.148438    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4h8s\" (UniqueName: \"kubernetes.io/projected/0f970ab0-becb-42e7-bdf0-e04a8267c531-kube-api-access-s4h8s\") pod \"maas-api-e2e-shared-a-e9225c-76cf7db755-rc2bb\" (UID: \"0f970ab0-becb-42e7-bdf0-e04a8267c531\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e9225c-76cf7db755-rc2bb"
Aug 02 10:21:22.148481 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:21:22.148468    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-e9225c-serving-cert: secret "maas-api-e2e-shared-a-e9225c-serving-cert" not found
Aug 02 10:21:22.148648 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:21:22.148542    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f970ab0-becb-42e7-bdf0-e04a8267c531-maas-api-tls podName:0f970ab0-becb-42e7-bdf0-e04a8267c531 nodeName:}" failed. No retries permitted until 2026-08-02 10:21:22.648523824 +0000 UTC m=+2906.363329260 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0f970ab0-becb-42e7-bdf0-e04a8267c531-maas-api-tls") pod "maas-api-e2e-shared-a-e9225c-76cf7db755-rc2bb" (UID: "0f970ab0-becb-42e7-bdf0-e04a8267c531") : secret "maas-api-e2e-shared-a-e9225c-serving-cert" not found
Aug 02 10:21:22.149021 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:22.149002    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f970ab0-becb-42e7-bdf0-e04a8267c531-openshift-service-ca\") pod \"maas-api-e2e-shared-a-e9225c-76cf7db755-rc2bb\" (UID: \"0f970ab0-becb-42e7-bdf0-e04a8267c531\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e9225c-76cf7db755-rc2bb"
Aug 02 10:21:22.158497 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:22.158475    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4h8s\" (UniqueName: \"kubernetes.io/projected/0f970ab0-becb-42e7-bdf0-e04a8267c531-kube-api-access-s4h8s\") pod \"maas-api-e2e-shared-a-e9225c-76cf7db755-rc2bb\" (UID: \"0f970ab0-becb-42e7-bdf0-e04a8267c531\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e9225c-76cf7db755-rc2bb"
Aug 02 10:21:22.328735 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:22.328638    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:21:22.521257 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:22.521201    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:21:22.620827 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:22.620747    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:21:22.652486 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:22.652452    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f970ab0-becb-42e7-bdf0-e04a8267c531-maas-api-tls\") pod \"maas-api-e2e-shared-a-e9225c-76cf7db755-rc2bb\" (UID: \"0f970ab0-becb-42e7-bdf0-e04a8267c531\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e9225c-76cf7db755-rc2bb"
Aug 02 10:21:22.653957 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:22.653933    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-e9225c-85b98c998-48zlf"]
Aug 02 10:21:22.654932 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:22.654909    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f970ab0-becb-42e7-bdf0-e04a8267c531-maas-api-tls\") pod \"maas-api-e2e-shared-a-e9225c-76cf7db755-rc2bb\" (UID: \"0f970ab0-becb-42e7-bdf0-e04a8267c531\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e9225c-76cf7db755-rc2bb"
Aug 02 10:21:22.657342 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:22.657327    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-e9225c-85b98c998-48zlf"
Aug 02 10:21:22.660137 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:22.660102    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-e9225c\""
Aug 02 10:21:22.660579 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:22.660432    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-e9225c-dockercfg-fs5k9\""
Aug 02 10:21:22.671499 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:22.671470    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-e9225c-85b98c998-48zlf"]
Aug 02 10:21:22.707159 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:22.707134    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-e9225c-7d9f9d5fb-pmc6f"]
Aug 02 10:21:22.710471 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:22.710453    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-e9225c-7d9f9d5fb-pmc6f"
Aug 02 10:21:22.719337 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:22.719318    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-e9225c-7d9f9d5fb-pmc6f"]
Aug 02 10:21:22.753929 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:22.753899    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0a74cc0-3669-4745-b2b9-e3af72f2671f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-e9225c-85b98c998-48zlf\" (UID: \"b0a74cc0-3669-4745-b2b9-e3af72f2671f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-e9225c-85b98c998-48zlf"
Aug 02 10:21:22.753929 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:22.753928    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl7cc\" (UniqueName: \"kubernetes.io/projected/b0a74cc0-3669-4745-b2b9-e3af72f2671f-kube-api-access-rl7cc\") pod \"payload-processing-e2e-shared-a-e9225c-85b98c998-48zlf\" (UID: \"b0a74cc0-3669-4745-b2b9-e3af72f2671f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-e9225c-85b98c998-48zlf"
Aug 02 10:21:22.854935 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:22.854907    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0a74cc0-3669-4745-b2b9-e3af72f2671f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-e9225c-85b98c998-48zlf\" (UID: \"b0a74cc0-3669-4745-b2b9-e3af72f2671f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-e9225c-85b98c998-48zlf"
Aug 02 10:21:22.855049 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:22.854940    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl7cc\" (UniqueName: \"kubernetes.io/projected/b0a74cc0-3669-4745-b2b9-e3af72f2671f-kube-api-access-rl7cc\") pod \"payload-processing-e2e-shared-a-e9225c-85b98c998-48zlf\" (UID: \"b0a74cc0-3669-4745-b2b9-e3af72f2671f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-e9225c-85b98c998-48zlf"
Aug 02 10:21:22.855049 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:22.854969    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71a2ede0-fd50-4c40-8076-047501f50391-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-e9225c-7d9f9d5fb-pmc6f\" (UID: \"71a2ede0-fd50-4c40-8076-047501f50391\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-e9225c-7d9f9d5fb-pmc6f"
Aug 02 10:21:22.855049 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:22.855007    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgmpv\" (UniqueName: \"kubernetes.io/projected/71a2ede0-fd50-4c40-8076-047501f50391-kube-api-access-bgmpv\") pod \"payload-pre-processing-e2e-shared-a-e9225c-7d9f9d5fb-pmc6f\" (UID: \"71a2ede0-fd50-4c40-8076-047501f50391\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-e9225c-7d9f9d5fb-pmc6f"
Aug 02 10:21:22.855625 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:22.855608    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0a74cc0-3669-4745-b2b9-e3af72f2671f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-e9225c-85b98c998-48zlf\" (UID: \"b0a74cc0-3669-4745-b2b9-e3af72f2671f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-e9225c-85b98c998-48zlf"
Aug 02 10:21:22.862857 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:22.862838    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl7cc\" (UniqueName: \"kubernetes.io/projected/b0a74cc0-3669-4745-b2b9-e3af72f2671f-kube-api-access-rl7cc\") pod \"payload-processing-e2e-shared-a-e9225c-85b98c998-48zlf\" (UID: \"b0a74cc0-3669-4745-b2b9-e3af72f2671f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-e9225c-85b98c998-48zlf"
Aug 02 10:21:22.900012 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:22.899990    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e9225c-76cf7db755-rc2bb"
Aug 02 10:21:22.955670 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:22.955637    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71a2ede0-fd50-4c40-8076-047501f50391-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-e9225c-7d9f9d5fb-pmc6f\" (UID: \"71a2ede0-fd50-4c40-8076-047501f50391\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-e9225c-7d9f9d5fb-pmc6f"
Aug 02 10:21:22.955865 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:22.955682    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgmpv\" (UniqueName: \"kubernetes.io/projected/71a2ede0-fd50-4c40-8076-047501f50391-kube-api-access-bgmpv\") pod \"payload-pre-processing-e2e-shared-a-e9225c-7d9f9d5fb-pmc6f\" (UID: \"71a2ede0-fd50-4c40-8076-047501f50391\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-e9225c-7d9f9d5fb-pmc6f"
Aug 02 10:21:22.956303 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:22.956280    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71a2ede0-fd50-4c40-8076-047501f50391-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-e9225c-7d9f9d5fb-pmc6f\" (UID: \"71a2ede0-fd50-4c40-8076-047501f50391\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-e9225c-7d9f9d5fb-pmc6f"
Aug 02 10:21:22.964597 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:22.964568    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgmpv\" (UniqueName: \"kubernetes.io/projected/71a2ede0-fd50-4c40-8076-047501f50391-kube-api-access-bgmpv\") pod \"payload-pre-processing-e2e-shared-a-e9225c-7d9f9d5fb-pmc6f\" (UID: \"71a2ede0-fd50-4c40-8076-047501f50391\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-e9225c-7d9f9d5fb-pmc6f"
Aug 02 10:21:22.970113 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:22.970090    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-e9225c-85b98c998-48zlf"
Aug 02 10:21:23.019691 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:23.019654    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-e9225c-7d9f9d5fb-pmc6f"
Aug 02 10:21:23.032046 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:23.032018    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e9225c-76cf7db755-rc2bb"]
Aug 02 10:21:23.034906 ip-10-0-141-232 kubenswrapper[2564]: W0802 10:21:23.034878    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f970ab0_becb_42e7_bdf0_e04a8267c531.slice/crio-fef3dff8f2cab841e36524a8072386d5cdc3653350b3cb2b0a8188752939ef80 WatchSource:0}: Error finding container fef3dff8f2cab841e36524a8072386d5cdc3653350b3cb2b0a8188752939ef80: Status 404 returned error can't find the container with id fef3dff8f2cab841e36524a8072386d5cdc3653350b3cb2b0a8188752939ef80
Aug 02 10:21:23.108607 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:23.108511    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-e9225c-85b98c998-48zlf"]
Aug 02 10:21:23.112058 ip-10-0-141-232 kubenswrapper[2564]: W0802 10:21:23.112034    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0a74cc0_3669_4745_b2b9_e3af72f2671f.slice/crio-cce5918404f182f0d3cdfd78a7f0884ff7d4f7c8857a83d63f0e77da1eb3112f WatchSource:0}: Error finding container cce5918404f182f0d3cdfd78a7f0884ff7d4f7c8857a83d63f0e77da1eb3112f: Status 404 returned error can't find the container with id cce5918404f182f0d3cdfd78a7f0884ff7d4f7c8857a83d63f0e77da1eb3112f
Aug 02 10:21:23.155358 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:23.155319    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-e9225c-7d9f9d5fb-pmc6f"]
Aug 02 10:21:23.157979 ip-10-0-141-232 kubenswrapper[2564]: W0802 10:21:23.157941    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71a2ede0_fd50_4c40_8076_047501f50391.slice/crio-5435e62ef333c40dac07942653d963ece0628ab0a6aa6ac7cf94df9a25ef7613 WatchSource:0}: Error finding container 5435e62ef333c40dac07942653d963ece0628ab0a6aa6ac7cf94df9a25ef7613: Status 404 returned error can't find the container with id 5435e62ef333c40dac07942653d963ece0628ab0a6aa6ac7cf94df9a25ef7613
Aug 02 10:21:23.343726 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:23.343693    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-e9225c-85b98c998-48zlf" event={"ID":"b0a74cc0-3669-4745-b2b9-e3af72f2671f","Type":"ContainerStarted","Data":"84b82df9a78d886fa0d4aede6b7318e7b64a325a1d3ebf87a7eeb47ff3fc7262"}
Aug 02 10:21:23.343908 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:23.343829    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-e9225c-85b98c998-48zlf"
Aug 02 10:21:23.343908 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:23.343864    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-e9225c-85b98c998-48zlf" event={"ID":"b0a74cc0-3669-4745-b2b9-e3af72f2671f","Type":"ContainerStarted","Data":"cce5918404f182f0d3cdfd78a7f0884ff7d4f7c8857a83d63f0e77da1eb3112f"}
Aug 02 10:21:23.345380 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:23.345305    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-e9225c-7d9f9d5fb-pmc6f" event={"ID":"71a2ede0-fd50-4c40-8076-047501f50391","Type":"ContainerStarted","Data":"c221aa89fa3b24952d6b4178544e65506076c508b8d79bfae981bdf57a26bf68"}
Aug 02 10:21:23.345380 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:23.345350    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-e9225c-7d9f9d5fb-pmc6f" event={"ID":"71a2ede0-fd50-4c40-8076-047501f50391","Type":"ContainerStarted","Data":"5435e62ef333c40dac07942653d963ece0628ab0a6aa6ac7cf94df9a25ef7613"}
Aug 02 10:21:23.345596 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:23.345428    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-e9225c-7d9f9d5fb-pmc6f"
Aug 02 10:21:23.346738 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:23.346704    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e9225c-76cf7db755-rc2bb" event={"ID":"0f970ab0-becb-42e7-bdf0-e04a8267c531","Type":"ContainerStarted","Data":"fef3dff8f2cab841e36524a8072386d5cdc3653350b3cb2b0a8188752939ef80"}
Aug 02 10:21:23.362253 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:23.362203    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-e9225c-85b98c998-48zlf" podStartSLOduration=1.36219173 podStartE2EDuration="1.36219173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:21:23.360809216 +0000 UTC m=+2907.075614687" watchObservedRunningTime="2026-08-02 10:21:23.36219173 +0000 UTC m=+2907.076997178"
Aug 02 10:21:23.376668 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:23.376619    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-e9225c-7d9f9d5fb-pmc6f" podStartSLOduration=1.376601609 podStartE2EDuration="1.376601609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:21:23.375153675 +0000 UTC m=+2907.089959116" watchObservedRunningTime="2026-08-02 10:21:23.376601609 +0000 UTC m=+2907.091407096"
Aug 02 10:21:24.352298 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:24.352261    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e9225c-76cf7db755-rc2bb" event={"ID":"0f970ab0-becb-42e7-bdf0-e04a8267c531","Type":"ContainerStarted","Data":"7d5123b7d75e1d26e20dc9cb221b82347a1cbf2fea10dd0cfb8b4d180ce19a45"}
Aug 02 10:21:24.352736 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:24.352654    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e9225c-76cf7db755-rc2bb"
Aug 02 10:21:24.369899 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:24.369844    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e9225c-76cf7db755-rc2bb" podStartSLOduration=3.026365708 podStartE2EDuration="3.369826491s" totalImagesPullingTime="343.460783ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:21:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:21:24.367839353 +0000 UTC m=+2908.082644804" watchObservedRunningTime="2026-08-02 10:21:24.369826491 +0000 UTC m=+2908.084631943"
Aug 02 10:21:26.658654 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:26.658614    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:21:31.367300 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:31.367271    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e9225c-76cf7db755-rc2bb"
Aug 02 10:21:34.354002 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:34.353958    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-e9225c-85b98c998-48zlf"
Aug 02 10:21:34.354443 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:34.354029    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-e9225c-7d9f9d5fb-pmc6f"
Aug 02 10:21:45.910072 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:45.910035    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:21:46.255224 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:46.255142    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:21:46.599750 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:46.599666    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:21:47.543108 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:47.543075    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:21:49.041082 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:49.041055    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:21:49.122657 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:49.122619    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:21:50.128265 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:50.128224    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-0c6483-bc76ffcb6-fvz7z"]
Aug 02 10:21:50.131729 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:50.131693    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0c6483-bc76ffcb6-fvz7z"
Aug 02 10:21:50.134149 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:50.134131    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-0c6483-serving-cert\""
Aug 02 10:21:50.142685 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:50.142663    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-0c6483-bc76ffcb6-fvz7z"]
Aug 02 10:21:50.166132 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:50.166108    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:21:50.266049 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:50.266016    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14dd43af-5b06-41c4-9f85-bd3e4eae7f5b-maas-api-tls\") pod \"maas-api-e2e-shared-b-0c6483-bc76ffcb6-fvz7z\" (UID: \"14dd43af-5b06-41c4-9f85-bd3e4eae7f5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0c6483-bc76ffcb6-fvz7z"
Aug 02 10:21:50.266215 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:50.266072    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14dd43af-5b06-41c4-9f85-bd3e4eae7f5b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-0c6483-bc76ffcb6-fvz7z\" (UID: \"14dd43af-5b06-41c4-9f85-bd3e4eae7f5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0c6483-bc76ffcb6-fvz7z"
Aug 02 10:21:50.266215 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:50.266096    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqxvb\" (UniqueName: \"kubernetes.io/projected/14dd43af-5b06-41c4-9f85-bd3e4eae7f5b-kube-api-access-qqxvb\") pod \"maas-api-e2e-shared-b-0c6483-bc76ffcb6-fvz7z\" (UID: \"14dd43af-5b06-41c4-9f85-bd3e4eae7f5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0c6483-bc76ffcb6-fvz7z"
Aug 02 10:21:50.340989 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:50.340959    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:21:50.367080 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:50.367051    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14dd43af-5b06-41c4-9f85-bd3e4eae7f5b-maas-api-tls\") pod \"maas-api-e2e-shared-b-0c6483-bc76ffcb6-fvz7z\" (UID: \"14dd43af-5b06-41c4-9f85-bd3e4eae7f5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0c6483-bc76ffcb6-fvz7z"
Aug 02 10:21:50.367217 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:50.367088    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14dd43af-5b06-41c4-9f85-bd3e4eae7f5b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-0c6483-bc76ffcb6-fvz7z\" (UID: \"14dd43af-5b06-41c4-9f85-bd3e4eae7f5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0c6483-bc76ffcb6-fvz7z"
Aug 02 10:21:50.367217 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:50.367107    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqxvb\" (UniqueName: \"kubernetes.io/projected/14dd43af-5b06-41c4-9f85-bd3e4eae7f5b-kube-api-access-qqxvb\") pod \"maas-api-e2e-shared-b-0c6483-bc76ffcb6-fvz7z\" (UID: \"14dd43af-5b06-41c4-9f85-bd3e4eae7f5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0c6483-bc76ffcb6-fvz7z"
Aug 02 10:21:50.367814 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:50.367793    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14dd43af-5b06-41c4-9f85-bd3e4eae7f5b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-0c6483-bc76ffcb6-fvz7z\" (UID: \"14dd43af-5b06-41c4-9f85-bd3e4eae7f5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0c6483-bc76ffcb6-fvz7z"
Aug 02 10:21:50.369611 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:50.369589    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14dd43af-5b06-41c4-9f85-bd3e4eae7f5b-maas-api-tls\") pod \"maas-api-e2e-shared-b-0c6483-bc76ffcb6-fvz7z\" (UID: \"14dd43af-5b06-41c4-9f85-bd3e4eae7f5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0c6483-bc76ffcb6-fvz7z"
Aug 02 10:21:50.374752 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:50.374730    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqxvb\" (UniqueName: \"kubernetes.io/projected/14dd43af-5b06-41c4-9f85-bd3e4eae7f5b-kube-api-access-qqxvb\") pod \"maas-api-e2e-shared-b-0c6483-bc76ffcb6-fvz7z\" (UID: \"14dd43af-5b06-41c4-9f85-bd3e4eae7f5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0c6483-bc76ffcb6-fvz7z"
Aug 02 10:21:50.443977 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:50.443921    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0c6483-bc76ffcb6-fvz7z"
Aug 02 10:21:50.572935 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:50.572912    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-0c6483-bc76ffcb6-fvz7z"]
Aug 02 10:21:50.575190 ip-10-0-141-232 kubenswrapper[2564]: W0802 10:21:50.575160    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14dd43af_5b06_41c4_9f85_bd3e4eae7f5b.slice/crio-9c8e5edeb87131d8ffeb3da9ad3bd7cd18e22c82b89402dc4203e0391dc2aa6b WatchSource:0}: Error finding container 9c8e5edeb87131d8ffeb3da9ad3bd7cd18e22c82b89402dc4203e0391dc2aa6b: Status 404 returned error can't find the container with id 9c8e5edeb87131d8ffeb3da9ad3bd7cd18e22c82b89402dc4203e0391dc2aa6b
Aug 02 10:21:50.576548 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:50.576527    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 02 10:21:50.943372 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:50.942519    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:21:51.034907 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:51.034868    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:21:51.054245 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:51.054202    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-0c6483-dfc8557fb-wtg6n"]
Aug 02 10:21:51.058840 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:51.058821    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-0c6483-dfc8557fb-wtg6n"
Aug 02 10:21:51.061202 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:51.061186    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-0c6483-dockercfg-ff5jz\""
Aug 02 10:21:51.061310 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:51.061261    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-0c6483\""
Aug 02 10:21:51.065258 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:51.065211    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-0c6483-dfc8557fb-wtg6n"]
Aug 02 10:21:51.108943 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:51.108907    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-0c6483-59bf4c67c4-mcv6f"]
Aug 02 10:21:51.112094 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:51.112074    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-0c6483-59bf4c67c4-mcv6f"
Aug 02 10:21:51.127886 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:51.127863    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-0c6483-59bf4c67c4-mcv6f"]
Aug 02 10:21:51.173684 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:51.173656    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lzmt\" (UniqueName: \"kubernetes.io/projected/5284a7de-f4d1-4f6a-9f6d-94e8b6926527-kube-api-access-4lzmt\") pod \"payload-pre-processing-e2e-shared-b-0c6483-dfc8557fb-wtg6n\" (UID: \"5284a7de-f4d1-4f6a-9f6d-94e8b6926527\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-0c6483-dfc8557fb-wtg6n"
Aug 02 10:21:51.173975 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:51.173772    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5284a7de-f4d1-4f6a-9f6d-94e8b6926527-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-0c6483-dfc8557fb-wtg6n\" (UID: \"5284a7de-f4d1-4f6a-9f6d-94e8b6926527\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-0c6483-dfc8557fb-wtg6n"
Aug 02 10:21:51.274706 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:51.274670    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bbc1b38-00d1-4004-b341-1b7bb0c8e2b6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-0c6483-59bf4c67c4-mcv6f\" (UID: \"6bbc1b38-00d1-4004-b341-1b7bb0c8e2b6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-0c6483-59bf4c67c4-mcv6f"
Aug 02 10:21:51.274867 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:51.274730    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5284a7de-f4d1-4f6a-9f6d-94e8b6926527-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-0c6483-dfc8557fb-wtg6n\" (UID: \"5284a7de-f4d1-4f6a-9f6d-94e8b6926527\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-0c6483-dfc8557fb-wtg6n"
Aug 02 10:21:51.274867 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:51.274784    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msbvm\" (UniqueName: \"kubernetes.io/projected/6bbc1b38-00d1-4004-b341-1b7bb0c8e2b6-kube-api-access-msbvm\") pod \"payload-processing-e2e-shared-b-0c6483-59bf4c67c4-mcv6f\" (UID: \"6bbc1b38-00d1-4004-b341-1b7bb0c8e2b6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-0c6483-59bf4c67c4-mcv6f"
Aug 02 10:21:51.274939 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:51.274874    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lzmt\" (UniqueName: \"kubernetes.io/projected/5284a7de-f4d1-4f6a-9f6d-94e8b6926527-kube-api-access-4lzmt\") pod \"payload-pre-processing-e2e-shared-b-0c6483-dfc8557fb-wtg6n\" (UID: \"5284a7de-f4d1-4f6a-9f6d-94e8b6926527\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-0c6483-dfc8557fb-wtg6n"
Aug 02 10:21:51.275401 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:51.275385    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5284a7de-f4d1-4f6a-9f6d-94e8b6926527-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-0c6483-dfc8557fb-wtg6n\" (UID: \"5284a7de-f4d1-4f6a-9f6d-94e8b6926527\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-0c6483-dfc8557fb-wtg6n"
Aug 02 10:21:51.283603 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:51.283587    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lzmt\" (UniqueName: \"kubernetes.io/projected/5284a7de-f4d1-4f6a-9f6d-94e8b6926527-kube-api-access-4lzmt\") pod \"payload-pre-processing-e2e-shared-b-0c6483-dfc8557fb-wtg6n\" (UID: \"5284a7de-f4d1-4f6a-9f6d-94e8b6926527\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-0c6483-dfc8557fb-wtg6n"
Aug 02 10:21:51.370985 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:51.370950    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-0c6483-dfc8557fb-wtg6n"
Aug 02 10:21:51.375315 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:51.375289    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msbvm\" (UniqueName: \"kubernetes.io/projected/6bbc1b38-00d1-4004-b341-1b7bb0c8e2b6-kube-api-access-msbvm\") pod \"payload-processing-e2e-shared-b-0c6483-59bf4c67c4-mcv6f\" (UID: \"6bbc1b38-00d1-4004-b341-1b7bb0c8e2b6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-0c6483-59bf4c67c4-mcv6f"
Aug 02 10:21:51.375440 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:51.375358    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bbc1b38-00d1-4004-b341-1b7bb0c8e2b6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-0c6483-59bf4c67c4-mcv6f\" (UID: \"6bbc1b38-00d1-4004-b341-1b7bb0c8e2b6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-0c6483-59bf4c67c4-mcv6f"
Aug 02 10:21:51.376095 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:51.376073    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bbc1b38-00d1-4004-b341-1b7bb0c8e2b6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-0c6483-59bf4c67c4-mcv6f\" (UID: \"6bbc1b38-00d1-4004-b341-1b7bb0c8e2b6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-0c6483-59bf4c67c4-mcv6f"
Aug 02 10:21:51.389539 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:51.389514    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msbvm\" (UniqueName: \"kubernetes.io/projected/6bbc1b38-00d1-4004-b341-1b7bb0c8e2b6-kube-api-access-msbvm\") pod \"payload-processing-e2e-shared-b-0c6483-59bf4c67c4-mcv6f\" (UID: \"6bbc1b38-00d1-4004-b341-1b7bb0c8e2b6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-0c6483-59bf4c67c4-mcv6f"
Aug 02 10:21:51.422098 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:51.422065    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-0c6483-59bf4c67c4-mcv6f"
Aug 02 10:21:51.466901 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:51.466455    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0c6483-bc76ffcb6-fvz7z" event={"ID":"14dd43af-5b06-41c4-9f85-bd3e4eae7f5b","Type":"ContainerStarted","Data":"579ebdcbab598b9502387e664aeaeb40de0bc841a1ff7a22eecda5b58330b614"}
Aug 02 10:21:51.466901 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:51.466521    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0c6483-bc76ffcb6-fvz7z" event={"ID":"14dd43af-5b06-41c4-9f85-bd3e4eae7f5b","Type":"ContainerStarted","Data":"9c8e5edeb87131d8ffeb3da9ad3bd7cd18e22c82b89402dc4203e0391dc2aa6b"}
Aug 02 10:21:51.466901 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:51.466574    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0c6483-bc76ffcb6-fvz7z"
Aug 02 10:21:51.509515 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:51.509455    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0c6483-bc76ffcb6-fvz7z" podStartSLOduration=0.944522844 podStartE2EDuration="1.509434363s" totalImagesPullingTime="564.911519ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:21:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:21:51.489072064 +0000 UTC m=+2935.203877513" watchObservedRunningTime="2026-08-02 10:21:51.509434363 +0000 UTC m=+2935.224239810"
Aug 02 10:21:51.511429 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:51.511403    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-0c6483-dfc8557fb-wtg6n"]
Aug 02 10:21:51.515840 ip-10-0-141-232 kubenswrapper[2564]: W0802 10:21:51.515631    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5284a7de_f4d1_4f6a_9f6d_94e8b6926527.slice/crio-c17542634f9708e16598d620aabbd33c5358a6c20f11aee7815dd4b3b675b59f WatchSource:0}: Error finding container c17542634f9708e16598d620aabbd33c5358a6c20f11aee7815dd4b3b675b59f: Status 404 returned error can't find the container with id c17542634f9708e16598d620aabbd33c5358a6c20f11aee7815dd4b3b675b59f
Aug 02 10:21:51.567548 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:51.567525    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-0c6483-59bf4c67c4-mcv6f"]
Aug 02 10:21:51.569369 ip-10-0-141-232 kubenswrapper[2564]: W0802 10:21:51.569342    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bbc1b38_00d1_4004_b341_1b7bb0c8e2b6.slice/crio-d0fb4c1de81e51eb3289008acad5f6c7e5c8f81142d767449efd922ae1f407db WatchSource:0}: Error finding container d0fb4c1de81e51eb3289008acad5f6c7e5c8f81142d767449efd922ae1f407db: Status 404 returned error can't find the container with id d0fb4c1de81e51eb3289008acad5f6c7e5c8f81142d767449efd922ae1f407db
Aug 02 10:21:52.472303 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:52.472263    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-0c6483-59bf4c67c4-mcv6f" event={"ID":"6bbc1b38-00d1-4004-b341-1b7bb0c8e2b6","Type":"ContainerStarted","Data":"557d6540a0dc82f2ae292943bd2939d0124737a00e4889abf3e90cd364adf7d3"}
Aug 02 10:21:52.472880 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:52.472326    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-0c6483-59bf4c67c4-mcv6f" event={"ID":"6bbc1b38-00d1-4004-b341-1b7bb0c8e2b6","Type":"ContainerStarted","Data":"d0fb4c1de81e51eb3289008acad5f6c7e5c8f81142d767449efd922ae1f407db"}
Aug 02 10:21:52.472880 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:52.472455    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-0c6483-59bf4c67c4-mcv6f"
Aug 02 10:21:52.473745 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:52.473724    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-0c6483-dfc8557fb-wtg6n" event={"ID":"5284a7de-f4d1-4f6a-9f6d-94e8b6926527","Type":"ContainerStarted","Data":"29a593efb17af90016cfcaf4b6c706c6a25856c961de8330cf68210ea477ced2"}
Aug 02 10:21:52.473847 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:52.473767    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-0c6483-dfc8557fb-wtg6n" event={"ID":"5284a7de-f4d1-4f6a-9f6d-94e8b6926527","Type":"ContainerStarted","Data":"c17542634f9708e16598d620aabbd33c5358a6c20f11aee7815dd4b3b675b59f"}
Aug 02 10:21:52.488591 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:52.488548    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-0c6483-59bf4c67c4-mcv6f" podStartSLOduration=1.488535029 podStartE2EDuration="1.488535029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:21:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:21:52.487539018 +0000 UTC m=+2936.202344467" watchObservedRunningTime="2026-08-02 10:21:52.488535029 +0000 UTC m=+2936.203340476"
Aug 02 10:21:52.505036 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:52.504998    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-0c6483-dfc8557fb-wtg6n" podStartSLOduration=1.5049863239999999 podStartE2EDuration="1.504986324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:21:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:21:52.50337736 +0000 UTC m=+2936.218182809" watchObservedRunningTime="2026-08-02 10:21:52.504986324 +0000 UTC m=+2936.219791771"
Aug 02 10:21:53.477839 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:53.477808    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-0c6483-dfc8557fb-wtg6n"
Aug 02 10:21:57.479274 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:21:57.479229    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0c6483-bc76ffcb6-fvz7z"
Aug 02 10:22:03.479584 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:22:03.479556    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-0c6483-59bf4c67c4-mcv6f"
Aug 02 10:22:04.484369 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:22:04.484337    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-0c6483-dfc8557fb-wtg6n"
Aug 02 10:22:13.927701 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:22:13.927669    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:22:14.027755 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:22:14.027719    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:22:33.830467 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:22:33.830382    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:22:44.969152 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:22:44.969115    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:22:51.936296 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:22:51.936265    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:24:39.508985 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:24:39.508932    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:26:39.538463 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:26:39.538425    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:26:53.628556 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:26:53.628453    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:27:10.124572 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:27:10.124527    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:27:10.226865 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:27:10.226835    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:27:19.010105 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:27:19.010071    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:27:19.922728 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:27:19.922695    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:27:26.487064 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:27:26.487028    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-e9225c/sub-test-model-e9225c-kserve-788df4fc55-fkf2n"]
Aug 02 10:27:26.490605 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:27:26.490583    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e9225c/sub-test-model-e9225c-kserve-788df4fc55-fkf2n"
Aug 02 10:27:26.493730 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:27:26.493705    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-e9225c\"/\"kube-root-ca.crt\""
Aug 02 10:27:26.493960 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:27:26.493790    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e9225c\"/\"sub-test-model-e9225c-kserve-self-signed-certs\""
Aug 02 10:27:26.493960 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:27:26.493794    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-e9225c\"/\"openshift-service-ca.crt\""
Aug 02 10:27:26.494110 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:27:26.493850    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e9225c\"/\"default-dockercfg-mxshh\""
Aug 02 10:27:26.500063 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:27:26.500040    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e9225c/sub-test-model-e9225c-kserve-788df4fc55-fkf2n"]
Aug 02 10:27:26.510552 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:27:26.510519    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:27:26.600032 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:27:26.600000    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acd0bfe9-42d7-448b-93dc-755e03983f90-model-cache\") pod \"sub-test-model-e9225c-kserve-788df4fc55-fkf2n\" (UID: \"acd0bfe9-42d7-448b-93dc-755e03983f90\") " pod="ai-tenant-e2e-shared-a-e9225c/sub-test-model-e9225c-kserve-788df4fc55-fkf2n"
Aug 02 10:27:26.600201 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:27:26.600061    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acd0bfe9-42d7-448b-93dc-755e03983f90-tls-certs\") pod \"sub-test-model-e9225c-kserve-788df4fc55-fkf2n\" (UID: \"acd0bfe9-42d7-448b-93dc-755e03983f90\") " pod="ai-tenant-e2e-shared-a-e9225c/sub-test-model-e9225c-kserve-788df4fc55-fkf2n"
Aug 02 10:27:26.600201 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:27:26.600096    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjnz9\" (UniqueName: \"kubernetes.io/projected/acd0bfe9-42d7-448b-93dc-755e03983f90-kube-api-access-bjnz9\") pod \"sub-test-model-e9225c-kserve-788df4fc55-fkf2n\" (UID: \"acd0bfe9-42d7-448b-93dc-755e03983f90\") " pod="ai-tenant-e2e-shared-a-e9225c/sub-test-model-e9225c-kserve-788df4fc55-fkf2n"
Aug 02 10:27:26.600201 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:27:26.600128    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acd0bfe9-42d7-448b-93dc-755e03983f90-dshm\") pod \"sub-test-model-e9225c-kserve-788df4fc55-fkf2n\" (UID: \"acd0bfe9-42d7-448b-93dc-755e03983f90\") " pod="ai-tenant-e2e-shared-a-e9225c/sub-test-model-e9225c-kserve-788df4fc55-fkf2n"
Aug 02 10:27:26.600201 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:27:26.600172    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acd0bfe9-42d7-448b-93dc-755e03983f90-home\") pod \"sub-test-model-e9225c-kserve-788df4fc55-fkf2n\" (UID: \"acd0bfe9-42d7-448b-93dc-755e03983f90\") " pod="ai-tenant-e2e-shared-a-e9225c/sub-test-model-e9225c-kserve-788df4fc55-fkf2n"
Aug 02 10:27:26.600201 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:27:26.600193    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acd0bfe9-42d7-448b-93dc-755e03983f90-tmp-dir\") pod \"sub-test-model-e9225c-kserve-788df4fc55-fkf2n\" (UID: \"acd0bfe9-42d7-448b-93dc-755e03983f90\") " pod="ai-tenant-e2e-shared-a-e9225c/sub-test-model-e9225c-kserve-788df4fc55-fkf2n"
Aug 02 10:27:26.701481 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:27:26.701443    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acd0bfe9-42d7-448b-93dc-755e03983f90-home\") pod \"sub-test-model-e9225c-kserve-788df4fc55-fkf2n\" (UID: \"acd0bfe9-42d7-448b-93dc-755e03983f90\") " pod="ai-tenant-e2e-shared-a-e9225c/sub-test-model-e9225c-kserve-788df4fc55-fkf2n"
Aug 02 10:27:26.701653 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:27:26.701486    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acd0bfe9-42d7-448b-93dc-755e03983f90-tmp-dir\") pod \"sub-test-model-e9225c-kserve-788df4fc55-fkf2n\" (UID: \"acd0bfe9-42d7-448b-93dc-755e03983f90\") " pod="ai-tenant-e2e-shared-a-e9225c/sub-test-model-e9225c-kserve-788df4fc55-fkf2n"
Aug 02 10:27:26.701653 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:27:26.701537    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acd0bfe9-42d7-448b-93dc-755e03983f90-model-cache\") pod \"sub-test-model-e9225c-kserve-788df4fc55-fkf2n\" (UID: \"acd0bfe9-42d7-448b-93dc-755e03983f90\") " pod="ai-tenant-e2e-shared-a-e9225c/sub-test-model-e9225c-kserve-788df4fc55-fkf2n"
Aug 02 10:27:26.701653 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:27:26.701623    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acd0bfe9-42d7-448b-93dc-755e03983f90-tls-certs\") pod \"sub-test-model-e9225c-kserve-788df4fc55-fkf2n\" (UID: \"acd0bfe9-42d7-448b-93dc-755e03983f90\") " pod="ai-tenant-e2e-shared-a-e9225c/sub-test-model-e9225c-kserve-788df4fc55-fkf2n"
Aug 02 10:27:26.701653 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:27:26.701646    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjnz9\" (UniqueName: \"kubernetes.io/projected/acd0bfe9-42d7-448b-93dc-755e03983f90-kube-api-access-bjnz9\") pod \"sub-test-model-e9225c-kserve-788df4fc55-fkf2n\" (UID: \"acd0bfe9-42d7-448b-93dc-755e03983f90\") " pod="ai-tenant-e2e-shared-a-e9225c/sub-test-model-e9225c-kserve-788df4fc55-fkf2n"
Aug 02 10:27:26.701883 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:27:26.701673    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acd0bfe9-42d7-448b-93dc-755e03983f90-dshm\") pod \"sub-test-model-e9225c-kserve-788df4fc55-fkf2n\" (UID: \"acd0bfe9-42d7-448b-93dc-755e03983f90\") " pod="ai-tenant-e2e-shared-a-e9225c/sub-test-model-e9225c-kserve-788df4fc55-fkf2n"
Aug 02 10:27:26.702092 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:27:26.702062    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acd0bfe9-42d7-448b-93dc-755e03983f90-tmp-dir\") pod \"sub-test-model-e9225c-kserve-788df4fc55-fkf2n\" (UID: \"acd0bfe9-42d7-448b-93dc-755e03983f90\") " pod="ai-tenant-e2e-shared-a-e9225c/sub-test-model-e9225c-kserve-788df4fc55-fkf2n"
Aug 02 10:27:26.702323 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:27:26.702300    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acd0bfe9-42d7-448b-93dc-755e03983f90-home\") pod \"sub-test-model-e9225c-kserve-788df4fc55-fkf2n\" (UID: \"acd0bfe9-42d7-448b-93dc-755e03983f90\") " pod="ai-tenant-e2e-shared-a-e9225c/sub-test-model-e9225c-kserve-788df4fc55-fkf2n"
Aug 02 10:27:26.702439 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:27:26.702412    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acd0bfe9-42d7-448b-93dc-755e03983f90-model-cache\") pod \"sub-test-model-e9225c-kserve-788df4fc55-fkf2n\" (UID: \"acd0bfe9-42d7-448b-93dc-755e03983f90\") " pod="ai-tenant-e2e-shared-a-e9225c/sub-test-model-e9225c-kserve-788df4fc55-fkf2n"
Aug 02 10:27:26.703987 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:27:26.703967    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acd0bfe9-42d7-448b-93dc-755e03983f90-dshm\") pod \"sub-test-model-e9225c-kserve-788df4fc55-fkf2n\" (UID: \"acd0bfe9-42d7-448b-93dc-755e03983f90\") " pod="ai-tenant-e2e-shared-a-e9225c/sub-test-model-e9225c-kserve-788df4fc55-fkf2n"
Aug 02 10:27:26.704490 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:27:26.704472    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acd0bfe9-42d7-448b-93dc-755e03983f90-tls-certs\") pod \"sub-test-model-e9225c-kserve-788df4fc55-fkf2n\" (UID: \"acd0bfe9-42d7-448b-93dc-755e03983f90\") " pod="ai-tenant-e2e-shared-a-e9225c/sub-test-model-e9225c-kserve-788df4fc55-fkf2n"
Aug 02 10:27:26.709653 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:27:26.709632    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjnz9\" (UniqueName: \"kubernetes.io/projected/acd0bfe9-42d7-448b-93dc-755e03983f90-kube-api-access-bjnz9\") pod \"sub-test-model-e9225c-kserve-788df4fc55-fkf2n\" (UID: \"acd0bfe9-42d7-448b-93dc-755e03983f90\") " pod="ai-tenant-e2e-shared-a-e9225c/sub-test-model-e9225c-kserve-788df4fc55-fkf2n"
Aug 02 10:27:26.799488 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:27:26.799429    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e9225c/sub-test-model-e9225c-kserve-788df4fc55-fkf2n"
Aug 02 10:27:26.930542 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:27:26.930513    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e9225c/sub-test-model-e9225c-kserve-788df4fc55-fkf2n"]
Aug 02 10:27:26.933338 ip-10-0-141-232 kubenswrapper[2564]: W0802 10:27:26.933312    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podacd0bfe9_42d7_448b_93dc_755e03983f90.slice/crio-77227343428e28dc7ae53b8fe312e7bc756fc108a86bdaf61ea77dd2614f87ce WatchSource:0}: Error finding container 77227343428e28dc7ae53b8fe312e7bc756fc108a86bdaf61ea77dd2614f87ce: Status 404 returned error can't find the container with id 77227343428e28dc7ae53b8fe312e7bc756fc108a86bdaf61ea77dd2614f87ce
Aug 02 10:27:27.858988 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:27:27.858954    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e9225c/sub-test-model-e9225c-kserve-788df4fc55-fkf2n" event={"ID":"acd0bfe9-42d7-448b-93dc-755e03983f90","Type":"ContainerStarted","Data":"5456cce70f8885fdf8cabf8f0a4ed060a6883ccc04c722825efdb5c45c943286"}
Aug 02 10:27:27.859422 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:27:27.858997    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e9225c/sub-test-model-e9225c-kserve-788df4fc55-fkf2n" event={"ID":"acd0bfe9-42d7-448b-93dc-755e03983f90","Type":"ContainerStarted","Data":"77227343428e28dc7ae53b8fe312e7bc756fc108a86bdaf61ea77dd2614f87ce"}
Aug 02 10:27:27.877148 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:27:27.877098    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-e9225c/sub-test-model-e9225c-kserve-788df4fc55-fkf2n" podStartSLOduration=1.8770493240000001 podStartE2EDuration="1.877049324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:27:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:27:27.876535842 +0000 UTC m=+3271.591341290" watchObservedRunningTime="2026-08-02 10:27:27.877049324 +0000 UTC m=+3271.591854775"
Aug 02 10:27:36.800406 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:27:36.800364    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-e9225c/sub-test-model-e9225c-kserve-788df4fc55-fkf2n"
Aug 02 10:27:36.800406 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:27:36.800416    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-e9225c/sub-test-model-e9225c-kserve-788df4fc55-fkf2n"
Aug 02 10:27:36.813019 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:27:36.812992    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-e9225c/sub-test-model-e9225c-kserve-788df4fc55-fkf2n"
Aug 02 10:27:36.913578 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:27:36.913543    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-e9225c/sub-test-model-e9225c-kserve-788df4fc55-fkf2n"
Aug 02 10:28:29.925107 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:28:29.925070    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:29:28.923765 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:29:28.923717    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:29:38.799426 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:29:38.799395    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e9225c/sub-test-model-e9225c-kserve-788df4fc55-fkf2n"]
Aug 02 10:29:38.830727 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:29:38.830695    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:29:39.715628 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:29:39.715592    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:29:53.137150 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:29:53.137064    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:29:53.817770 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:29:53.817710    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-e9225c/sub-test-model-e9225c-kserve-788df4fc55-fkf2n" podUID="acd0bfe9-42d7-448b-93dc-755e03983f90" containerName="main" containerID="cri-o://5456cce70f8885fdf8cabf8f0a4ed060a6883ccc04c722825efdb5c45c943286" gracePeriod=45
Aug 02 10:29:54.063463 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:29:54.063437    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e9225c/sub-test-model-e9225c-kserve-788df4fc55-fkf2n"
Aug 02 10:29:54.103729 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:29:54.103648    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acd0bfe9-42d7-448b-93dc-755e03983f90-dshm\") pod \"acd0bfe9-42d7-448b-93dc-755e03983f90\" (UID: \"acd0bfe9-42d7-448b-93dc-755e03983f90\") "
Aug 02 10:29:54.103729 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:29:54.103703    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acd0bfe9-42d7-448b-93dc-755e03983f90-tmp-dir\") pod \"acd0bfe9-42d7-448b-93dc-755e03983f90\" (UID: \"acd0bfe9-42d7-448b-93dc-755e03983f90\") "
Aug 02 10:29:54.103952 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:29:54.103732    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjnz9\" (UniqueName: \"kubernetes.io/projected/acd0bfe9-42d7-448b-93dc-755e03983f90-kube-api-access-bjnz9\") pod \"acd0bfe9-42d7-448b-93dc-755e03983f90\" (UID: \"acd0bfe9-42d7-448b-93dc-755e03983f90\") "
Aug 02 10:29:54.103952 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:29:54.103777    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acd0bfe9-42d7-448b-93dc-755e03983f90-home\") pod \"acd0bfe9-42d7-448b-93dc-755e03983f90\" (UID: \"acd0bfe9-42d7-448b-93dc-755e03983f90\") "
Aug 02 10:29:54.103952 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:29:54.103807    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acd0bfe9-42d7-448b-93dc-755e03983f90-tls-certs\") pod \"acd0bfe9-42d7-448b-93dc-755e03983f90\" (UID: \"acd0bfe9-42d7-448b-93dc-755e03983f90\") "
Aug 02 10:29:54.103952 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:29:54.103863    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acd0bfe9-42d7-448b-93dc-755e03983f90-model-cache\") pod \"acd0bfe9-42d7-448b-93dc-755e03983f90\" (UID: \"acd0bfe9-42d7-448b-93dc-755e03983f90\") "
Aug 02 10:29:54.103952 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:29:54.103907    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acd0bfe9-42d7-448b-93dc-755e03983f90-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "acd0bfe9-42d7-448b-93dc-755e03983f90" (UID: "acd0bfe9-42d7-448b-93dc-755e03983f90"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 02 10:29:54.104264 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:29:54.104000    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acd0bfe9-42d7-448b-93dc-755e03983f90-home" (OuterVolumeSpecName: "home") pod "acd0bfe9-42d7-448b-93dc-755e03983f90" (UID: "acd0bfe9-42d7-448b-93dc-755e03983f90"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 02 10:29:54.104264 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:29:54.104139    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acd0bfe9-42d7-448b-93dc-755e03983f90-model-cache" (OuterVolumeSpecName: "model-cache") pod "acd0bfe9-42d7-448b-93dc-755e03983f90" (UID: "acd0bfe9-42d7-448b-93dc-755e03983f90"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 02 10:29:54.104264 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:29:54.104151    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acd0bfe9-42d7-448b-93dc-755e03983f90-home\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:29:54.104264 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:29:54.104170    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acd0bfe9-42d7-448b-93dc-755e03983f90-tmp-dir\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:29:54.105882 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:29:54.105854    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acd0bfe9-42d7-448b-93dc-755e03983f90-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "acd0bfe9-42d7-448b-93dc-755e03983f90" (UID: "acd0bfe9-42d7-448b-93dc-755e03983f90"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 02 10:29:54.105882 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:29:54.105872    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acd0bfe9-42d7-448b-93dc-755e03983f90-dshm" (OuterVolumeSpecName: "dshm") pod "acd0bfe9-42d7-448b-93dc-755e03983f90" (UID: "acd0bfe9-42d7-448b-93dc-755e03983f90"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 02 10:29:54.106060 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:29:54.105964    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acd0bfe9-42d7-448b-93dc-755e03983f90-kube-api-access-bjnz9" (OuterVolumeSpecName: "kube-api-access-bjnz9") pod "acd0bfe9-42d7-448b-93dc-755e03983f90" (UID: "acd0bfe9-42d7-448b-93dc-755e03983f90"). InnerVolumeSpecName "kube-api-access-bjnz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 02 10:29:54.205459 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:29:54.205413    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acd0bfe9-42d7-448b-93dc-755e03983f90-dshm\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:29:54.205459 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:29:54.205452    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjnz9\" (UniqueName: \"kubernetes.io/projected/acd0bfe9-42d7-448b-93dc-755e03983f90-kube-api-access-bjnz9\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:29:54.205459 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:29:54.205463    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acd0bfe9-42d7-448b-93dc-755e03983f90-tls-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:29:54.205459 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:29:54.205472    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acd0bfe9-42d7-448b-93dc-755e03983f90-model-cache\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:29:54.485752 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:29:54.485714    2564 generic.go:356] "Generic (PLEG): container finished" podID="acd0bfe9-42d7-448b-93dc-755e03983f90" containerID="5456cce70f8885fdf8cabf8f0a4ed060a6883ccc04c722825efdb5c45c943286" exitCode=0
Aug 02 10:29:54.485927 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:29:54.485762    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e9225c/sub-test-model-e9225c-kserve-788df4fc55-fkf2n"
Aug 02 10:29:54.485927 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:29:54.485802    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e9225c/sub-test-model-e9225c-kserve-788df4fc55-fkf2n" event={"ID":"acd0bfe9-42d7-448b-93dc-755e03983f90","Type":"ContainerDied","Data":"5456cce70f8885fdf8cabf8f0a4ed060a6883ccc04c722825efdb5c45c943286"}
Aug 02 10:29:54.485927 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:29:54.485843    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e9225c/sub-test-model-e9225c-kserve-788df4fc55-fkf2n" event={"ID":"acd0bfe9-42d7-448b-93dc-755e03983f90","Type":"ContainerDied","Data":"77227343428e28dc7ae53b8fe312e7bc756fc108a86bdaf61ea77dd2614f87ce"}
Aug 02 10:29:54.485927 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:29:54.485860    2564 scope.go:117] "RemoveContainer" containerID="5456cce70f8885fdf8cabf8f0a4ed060a6883ccc04c722825efdb5c45c943286"
Aug 02 10:29:54.494391 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:29:54.494369    2564 scope.go:117] "RemoveContainer" containerID="5456cce70f8885fdf8cabf8f0a4ed060a6883ccc04c722825efdb5c45c943286"
Aug 02 10:29:54.494611 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:29:54.494595    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5456cce70f8885fdf8cabf8f0a4ed060a6883ccc04c722825efdb5c45c943286\": container with ID starting with 5456cce70f8885fdf8cabf8f0a4ed060a6883ccc04c722825efdb5c45c943286 not found: ID does not exist" containerID="5456cce70f8885fdf8cabf8f0a4ed060a6883ccc04c722825efdb5c45c943286"
Aug 02 10:29:54.494676 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:29:54.494618    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5456cce70f8885fdf8cabf8f0a4ed060a6883ccc04c722825efdb5c45c943286"} err="failed to get container status \"5456cce70f8885fdf8cabf8f0a4ed060a6883ccc04c722825efdb5c45c943286\": rpc error: code = NotFound desc = could not find container \"5456cce70f8885fdf8cabf8f0a4ed060a6883ccc04c722825efdb5c45c943286\": container with ID starting with 5456cce70f8885fdf8cabf8f0a4ed060a6883ccc04c722825efdb5c45c943286 not found: ID does not exist"
Aug 02 10:29:54.514319 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:29:54.514294    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e9225c/sub-test-model-e9225c-kserve-788df4fc55-fkf2n"]
Aug 02 10:29:54.527458 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:29:54.527430    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-e9225c/sub-test-model-e9225c-kserve-788df4fc55-fkf2n"]
Aug 02 10:29:54.881127 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:29:54.881095    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acd0bfe9-42d7-448b-93dc-755e03983f90" path="/var/lib/kubelet/pods/acd0bfe9-42d7-448b-93dc-755e03983f90/volumes"
Aug 02 10:30:12.187336 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:30:12.187300    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:30:12.288725 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:30:12.288684    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:30:29.188068 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:30:29.188032    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:31:37.826929 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:31:37.826897    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:31:38.918345 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:31:38.918312    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:31:58.098393 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:31:58.098357    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:31:58.462539 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:31:58.462504    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:31:58.890580 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:31:58.890551    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:32:01.374175 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:01.374144    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:32:01.541291 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:01.541250    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:32:02.469242 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:02.469207    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c4717-c8c95bbbd-bvtlt"]
Aug 02 10:32:02.469768 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:02.469743    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acd0bfe9-42d7-448b-93dc-755e03983f90" containerName="main"
Aug 02 10:32:02.469768 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:02.469765    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="acd0bfe9-42d7-448b-93dc-755e03983f90" containerName="main"
Aug 02 10:32:02.469893 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:02.469851    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="acd0bfe9-42d7-448b-93dc-755e03983f90" containerName="main"
Aug 02 10:32:02.473074 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:02.473056    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c4717-c8c95bbbd-bvtlt"
Aug 02 10:32:02.475983 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:02.475959    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-5c4717-serving-cert\""
Aug 02 10:32:02.488624 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:02.488600    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c4717-c8c95bbbd-bvtlt"]
Aug 02 10:32:02.524135 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:02.524105    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:32:02.558133 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:02.558100    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzg4d\" (UniqueName: \"kubernetes.io/projected/62d25586-5c2b-4399-b1f2-e64d695d4aa6-kube-api-access-jzg4d\") pod \"maas-api-e2e-ipp-a-5c4717-c8c95bbbd-bvtlt\" (UID: \"62d25586-5c2b-4399-b1f2-e64d695d4aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c4717-c8c95bbbd-bvtlt"
Aug 02 10:32:02.558335 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:02.558176    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62d25586-5c2b-4399-b1f2-e64d695d4aa6-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5c4717-c8c95bbbd-bvtlt\" (UID: \"62d25586-5c2b-4399-b1f2-e64d695d4aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c4717-c8c95bbbd-bvtlt"
Aug 02 10:32:02.558335 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:02.558204    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62d25586-5c2b-4399-b1f2-e64d695d4aa6-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5c4717-c8c95bbbd-bvtlt\" (UID: \"62d25586-5c2b-4399-b1f2-e64d695d4aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c4717-c8c95bbbd-bvtlt"
Aug 02 10:32:02.659572 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:02.659529    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62d25586-5c2b-4399-b1f2-e64d695d4aa6-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5c4717-c8c95bbbd-bvtlt\" (UID: \"62d25586-5c2b-4399-b1f2-e64d695d4aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c4717-c8c95bbbd-bvtlt"
Aug 02 10:32:02.659572 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:02.659573    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62d25586-5c2b-4399-b1f2-e64d695d4aa6-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5c4717-c8c95bbbd-bvtlt\" (UID: \"62d25586-5c2b-4399-b1f2-e64d695d4aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c4717-c8c95bbbd-bvtlt"
Aug 02 10:32:02.659834 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:02.659609    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzg4d\" (UniqueName: \"kubernetes.io/projected/62d25586-5c2b-4399-b1f2-e64d695d4aa6-kube-api-access-jzg4d\") pod \"maas-api-e2e-ipp-a-5c4717-c8c95bbbd-bvtlt\" (UID: \"62d25586-5c2b-4399-b1f2-e64d695d4aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c4717-c8c95bbbd-bvtlt"
Aug 02 10:32:02.659834 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:32:02.659743    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c4717-serving-cert: secret "maas-api-e2e-ipp-a-5c4717-serving-cert" not found
Aug 02 10:32:02.659834 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:32:02.659817    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62d25586-5c2b-4399-b1f2-e64d695d4aa6-maas-api-tls podName:62d25586-5c2b-4399-b1f2-e64d695d4aa6 nodeName:}" failed. No retries permitted until 2026-08-02 10:32:03.159794999 +0000 UTC m=+3546.874600427 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/62d25586-5c2b-4399-b1f2-e64d695d4aa6-maas-api-tls") pod "maas-api-e2e-ipp-a-5c4717-c8c95bbbd-bvtlt" (UID: "62d25586-5c2b-4399-b1f2-e64d695d4aa6") : secret "maas-api-e2e-ipp-a-5c4717-serving-cert" not found
Aug 02 10:32:02.660328 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:02.660311    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62d25586-5c2b-4399-b1f2-e64d695d4aa6-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5c4717-c8c95bbbd-bvtlt\" (UID: \"62d25586-5c2b-4399-b1f2-e64d695d4aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c4717-c8c95bbbd-bvtlt"
Aug 02 10:32:02.670021 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:02.669998    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzg4d\" (UniqueName: \"kubernetes.io/projected/62d25586-5c2b-4399-b1f2-e64d695d4aa6-kube-api-access-jzg4d\") pod \"maas-api-e2e-ipp-a-5c4717-c8c95bbbd-bvtlt\" (UID: \"62d25586-5c2b-4399-b1f2-e64d695d4aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c4717-c8c95bbbd-bvtlt"
Aug 02 10:32:03.008736 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:03.008698    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:32:03.127782 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:03.127735    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:32:03.165026 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:03.164985    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62d25586-5c2b-4399-b1f2-e64d695d4aa6-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5c4717-c8c95bbbd-bvtlt\" (UID: \"62d25586-5c2b-4399-b1f2-e64d695d4aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c4717-c8c95bbbd-bvtlt"
Aug 02 10:32:03.167543 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:03.167516    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62d25586-5c2b-4399-b1f2-e64d695d4aa6-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5c4717-c8c95bbbd-bvtlt\" (UID: \"62d25586-5c2b-4399-b1f2-e64d695d4aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c4717-c8c95bbbd-bvtlt"
Aug 02 10:32:03.240797 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:03.240761    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:32:03.290931 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:03.290839    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5c4717-854ddd7664-nfp79"]
Aug 02 10:32:03.294352 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:03.294331    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c4717-854ddd7664-nfp79"
Aug 02 10:32:03.297099 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:03.297078    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-5c4717-dockercfg-9q6bd\""
Aug 02 10:32:03.297493 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:03.297477    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-5c4717\""
Aug 02 10:32:03.309047 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:03.309023    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5c4717-854ddd7664-nfp79"]
Aug 02 10:32:03.365955 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:03.365925    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54152ea7-799b-46d3-aa04-563e69cf9fb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5c4717-854ddd7664-nfp79\" (UID: \"54152ea7-799b-46d3-aa04-563e69cf9fb0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c4717-854ddd7664-nfp79"
Aug 02 10:32:03.366117 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:03.365961    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnzjk\" (UniqueName: \"kubernetes.io/projected/54152ea7-799b-46d3-aa04-563e69cf9fb0-kube-api-access-lnzjk\") pod \"payload-pre-processing-e2e-ipp-a-5c4717-854ddd7664-nfp79\" (UID: \"54152ea7-799b-46d3-aa04-563e69cf9fb0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c4717-854ddd7664-nfp79"
Aug 02 10:32:03.377608 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:03.377575    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5c4717-d6dcf56b8-7nbm4"]
Aug 02 10:32:03.380822 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:03.380806    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c4717-d6dcf56b8-7nbm4"
Aug 02 10:32:03.383008 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:03.382988    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c4717-c8c95bbbd-bvtlt"
Aug 02 10:32:03.403174 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:03.403149    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5c4717-d6dcf56b8-7nbm4"]
Aug 02 10:32:03.467304 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:03.467271    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db34d767-89e9-4806-80df-7f88b724297f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5c4717-d6dcf56b8-7nbm4\" (UID: \"db34d767-89e9-4806-80df-7f88b724297f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5c4717-d6dcf56b8-7nbm4"
Aug 02 10:32:03.467468 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:03.467332    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdbw7\" (UniqueName: \"kubernetes.io/projected/db34d767-89e9-4806-80df-7f88b724297f-kube-api-access-qdbw7\") pod \"payload-processing-e2e-ipp-a-5c4717-d6dcf56b8-7nbm4\" (UID: \"db34d767-89e9-4806-80df-7f88b724297f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5c4717-d6dcf56b8-7nbm4"
Aug 02 10:32:03.467468 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:03.467375    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54152ea7-799b-46d3-aa04-563e69cf9fb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5c4717-854ddd7664-nfp79\" (UID: \"54152ea7-799b-46d3-aa04-563e69cf9fb0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c4717-854ddd7664-nfp79"
Aug 02 10:32:03.467468 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:03.467428    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnzjk\" (UniqueName: \"kubernetes.io/projected/54152ea7-799b-46d3-aa04-563e69cf9fb0-kube-api-access-lnzjk\") pod \"payload-pre-processing-e2e-ipp-a-5c4717-854ddd7664-nfp79\" (UID: \"54152ea7-799b-46d3-aa04-563e69cf9fb0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c4717-854ddd7664-nfp79"
Aug 02 10:32:03.468046 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:03.468025    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54152ea7-799b-46d3-aa04-563e69cf9fb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5c4717-854ddd7664-nfp79\" (UID: \"54152ea7-799b-46d3-aa04-563e69cf9fb0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c4717-854ddd7664-nfp79"
Aug 02 10:32:03.477561 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:03.477535    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnzjk\" (UniqueName: \"kubernetes.io/projected/54152ea7-799b-46d3-aa04-563e69cf9fb0-kube-api-access-lnzjk\") pod \"payload-pre-processing-e2e-ipp-a-5c4717-854ddd7664-nfp79\" (UID: \"54152ea7-799b-46d3-aa04-563e69cf9fb0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c4717-854ddd7664-nfp79"
Aug 02 10:32:03.528727 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:03.528700    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c4717-c8c95bbbd-bvtlt"]
Aug 02 10:32:03.531184 ip-10-0-141-232 kubenswrapper[2564]: W0802 10:32:03.531159    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62d25586_5c2b_4399_b1f2_e64d695d4aa6.slice/crio-a19e5b96f2f97daf7f4bfa089be39be262816466b2c9ce41e1139991f41891e0 WatchSource:0}: Error finding container a19e5b96f2f97daf7f4bfa089be39be262816466b2c9ce41e1139991f41891e0: Status 404 returned error can't find the container with id a19e5b96f2f97daf7f4bfa089be39be262816466b2c9ce41e1139991f41891e0
Aug 02 10:32:03.532407 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:03.532387    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 02 10:32:03.568933 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:03.568873    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db34d767-89e9-4806-80df-7f88b724297f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5c4717-d6dcf56b8-7nbm4\" (UID: \"db34d767-89e9-4806-80df-7f88b724297f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5c4717-d6dcf56b8-7nbm4"
Aug 02 10:32:03.568933 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:03.568926    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdbw7\" (UniqueName: \"kubernetes.io/projected/db34d767-89e9-4806-80df-7f88b724297f-kube-api-access-qdbw7\") pod \"payload-processing-e2e-ipp-a-5c4717-d6dcf56b8-7nbm4\" (UID: \"db34d767-89e9-4806-80df-7f88b724297f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5c4717-d6dcf56b8-7nbm4"
Aug 02 10:32:03.569509 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:03.569492    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db34d767-89e9-4806-80df-7f88b724297f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5c4717-d6dcf56b8-7nbm4\" (UID: \"db34d767-89e9-4806-80df-7f88b724297f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5c4717-d6dcf56b8-7nbm4"
Aug 02 10:32:03.582852 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:03.582826    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdbw7\" (UniqueName: \"kubernetes.io/projected/db34d767-89e9-4806-80df-7f88b724297f-kube-api-access-qdbw7\") pod \"payload-processing-e2e-ipp-a-5c4717-d6dcf56b8-7nbm4\" (UID: \"db34d767-89e9-4806-80df-7f88b724297f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5c4717-d6dcf56b8-7nbm4"
Aug 02 10:32:03.605359 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:03.605334    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c4717-854ddd7664-nfp79"
Aug 02 10:32:03.693659 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:03.693626    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c4717-d6dcf56b8-7nbm4"
Aug 02 10:32:03.738380 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:03.738353    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5c4717-854ddd7664-nfp79"]
Aug 02 10:32:03.740256 ip-10-0-141-232 kubenswrapper[2564]: W0802 10:32:03.740210    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54152ea7_799b_46d3_aa04_563e69cf9fb0.slice/crio-b92f6d35e2f71970e74e5c659a7348595df554f87748f53fe5287993e941b62f WatchSource:0}: Error finding container b92f6d35e2f71970e74e5c659a7348595df554f87748f53fe5287993e941b62f: Status 404 returned error can't find the container with id b92f6d35e2f71970e74e5c659a7348595df554f87748f53fe5287993e941b62f
Aug 02 10:32:03.837640 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:03.837608    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5c4717-d6dcf56b8-7nbm4"]
Aug 02 10:32:03.839475 ip-10-0-141-232 kubenswrapper[2564]: W0802 10:32:03.839445    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb34d767_89e9_4806_80df_7f88b724297f.slice/crio-f8ea84460fb928328accc0d46fb99616c1bfb692001fa0c84652cd5d2ac9c00f WatchSource:0}: Error finding container f8ea84460fb928328accc0d46fb99616c1bfb692001fa0c84652cd5d2ac9c00f: Status 404 returned error can't find the container with id f8ea84460fb928328accc0d46fb99616c1bfb692001fa0c84652cd5d2ac9c00f
Aug 02 10:32:04.023299 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:04.023264    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c4717-c8c95bbbd-bvtlt" event={"ID":"62d25586-5c2b-4399-b1f2-e64d695d4aa6","Type":"ContainerStarted","Data":"48c968604963d144ce9c44c79a54f0e1c95f2bea03b16fa19921a58f68cb2293"}
Aug 02 10:32:04.023495 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:04.023324    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c4717-c8c95bbbd-bvtlt" event={"ID":"62d25586-5c2b-4399-b1f2-e64d695d4aa6","Type":"ContainerStarted","Data":"a19e5b96f2f97daf7f4bfa089be39be262816466b2c9ce41e1139991f41891e0"}
Aug 02 10:32:04.023495 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:04.023352    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c4717-c8c95bbbd-bvtlt"
Aug 02 10:32:04.024712 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:04.024685    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c4717-d6dcf56b8-7nbm4" event={"ID":"db34d767-89e9-4806-80df-7f88b724297f","Type":"ContainerStarted","Data":"13c2775513e8858d38c61757435da728dfa1bc3b5544f1b1eb48cc2b13137704"}
Aug 02 10:32:04.024842 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:04.024735    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c4717-d6dcf56b8-7nbm4"
Aug 02 10:32:04.024842 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:04.024755    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c4717-d6dcf56b8-7nbm4" event={"ID":"db34d767-89e9-4806-80df-7f88b724297f","Type":"ContainerStarted","Data":"f8ea84460fb928328accc0d46fb99616c1bfb692001fa0c84652cd5d2ac9c00f"}
Aug 02 10:32:04.026353 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:04.026327    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c4717-854ddd7664-nfp79" event={"ID":"54152ea7-799b-46d3-aa04-563e69cf9fb0","Type":"ContainerStarted","Data":"94d15a1a5a0d26f111a5f3da5ac1e291409a70029b9f76f53dc1e4bb872c25c1"}
Aug 02 10:32:04.026463 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:04.026372    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c4717-854ddd7664-nfp79" event={"ID":"54152ea7-799b-46d3-aa04-563e69cf9fb0","Type":"ContainerStarted","Data":"b92f6d35e2f71970e74e5c659a7348595df554f87748f53fe5287993e941b62f"}
Aug 02 10:32:04.026582 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:04.026552    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c4717-854ddd7664-nfp79"
Aug 02 10:32:04.055647 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:04.055601    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c4717-c8c95bbbd-bvtlt" podStartSLOduration=1.6923583089999998 podStartE2EDuration="2.055584828s" totalImagesPullingTime="363.226519ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:32:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:32:04.053477301 +0000 UTC m=+3547.768282759" watchObservedRunningTime="2026-08-02 10:32:04.055584828 +0000 UTC m=+3547.770390294"
Aug 02 10:32:04.113211 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:04.113111    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c4717-854ddd7664-nfp79" podStartSLOduration=1.113096737 podStartE2EDuration="1.113096737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:32:04.111331387 +0000 UTC m=+3547.826136857" watchObservedRunningTime="2026-08-02 10:32:04.113096737 +0000 UTC m=+3547.827902184"
Aug 02 10:32:04.113211 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:04.113191    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c4717-d6dcf56b8-7nbm4" podStartSLOduration=1.113184489 podStartE2EDuration="1.113184489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:32:04.090872965 +0000 UTC m=+3547.805678409" watchObservedRunningTime="2026-08-02 10:32:04.113184489 +0000 UTC m=+3547.827989937"
Aug 02 10:32:10.036686 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:10.036654    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c4717-c8c95bbbd-bvtlt"
Aug 02 10:32:15.032438 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:15.032408    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c4717-d6dcf56b8-7nbm4"
Aug 02 10:32:15.032862 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:15.032791    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c4717-854ddd7664-nfp79"
Aug 02 10:32:26.331951 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:26.331920    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:32:27.025373 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:27.025340    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:32:27.362781 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:27.362687    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:32:27.992697 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:27.992657    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:32:30.246463 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:30.246429    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:32:30.346150 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:30.346115    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:32:31.500732 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:31.500692    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c167c0-6ccddcb77d-czh4t"]
Aug 02 10:32:31.503923 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:31.503900    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c167c0-6ccddcb77d-czh4t"
Aug 02 10:32:31.506419 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:31.506396    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-c167c0-serving-cert\""
Aug 02 10:32:31.515379 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:31.515354    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c167c0-6ccddcb77d-czh4t"]
Aug 02 10:32:31.557469 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:31.557430    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:32:31.609824 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:31.609787    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a3c555e-1c47-44b0-ab37-469473e8e931-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c167c0-6ccddcb77d-czh4t\" (UID: \"7a3c555e-1c47-44b0-ab37-469473e8e931\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c167c0-6ccddcb77d-czh4t"
Aug 02 10:32:31.610144 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:31.610119    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a3c555e-1c47-44b0-ab37-469473e8e931-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c167c0-6ccddcb77d-czh4t\" (UID: \"7a3c555e-1c47-44b0-ab37-469473e8e931\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c167c0-6ccddcb77d-czh4t"
Aug 02 10:32:31.610409 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:31.610389    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df2nz\" (UniqueName: \"kubernetes.io/projected/7a3c555e-1c47-44b0-ab37-469473e8e931-kube-api-access-df2nz\") pod \"maas-api-e2e-ipp-b-c167c0-6ccddcb77d-czh4t\" (UID: \"7a3c555e-1c47-44b0-ab37-469473e8e931\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c167c0-6ccddcb77d-czh4t"
Aug 02 10:32:31.711969 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:31.711932    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df2nz\" (UniqueName: \"kubernetes.io/projected/7a3c555e-1c47-44b0-ab37-469473e8e931-kube-api-access-df2nz\") pod \"maas-api-e2e-ipp-b-c167c0-6ccddcb77d-czh4t\" (UID: \"7a3c555e-1c47-44b0-ab37-469473e8e931\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c167c0-6ccddcb77d-czh4t"
Aug 02 10:32:31.712175 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:31.711977    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a3c555e-1c47-44b0-ab37-469473e8e931-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c167c0-6ccddcb77d-czh4t\" (UID: \"7a3c555e-1c47-44b0-ab37-469473e8e931\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c167c0-6ccddcb77d-czh4t"
Aug 02 10:32:31.712175 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:31.712002    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a3c555e-1c47-44b0-ab37-469473e8e931-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c167c0-6ccddcb77d-czh4t\" (UID: \"7a3c555e-1c47-44b0-ab37-469473e8e931\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c167c0-6ccddcb77d-czh4t"
Aug 02 10:32:31.712175 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:32:31.712118    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-c167c0-serving-cert: secret "maas-api-e2e-ipp-b-c167c0-serving-cert" not found
Aug 02 10:32:31.712367 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:32:31.712190    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a3c555e-1c47-44b0-ab37-469473e8e931-maas-api-tls podName:7a3c555e-1c47-44b0-ab37-469473e8e931 nodeName:}" failed. No retries permitted until 2026-08-02 10:32:32.212170918 +0000 UTC m=+3575.926976345 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7a3c555e-1c47-44b0-ab37-469473e8e931-maas-api-tls") pod "maas-api-e2e-ipp-b-c167c0-6ccddcb77d-czh4t" (UID: "7a3c555e-1c47-44b0-ab37-469473e8e931") : secret "maas-api-e2e-ipp-b-c167c0-serving-cert" not found
Aug 02 10:32:31.712617 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:31.712595    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a3c555e-1c47-44b0-ab37-469473e8e931-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c167c0-6ccddcb77d-czh4t\" (UID: \"7a3c555e-1c47-44b0-ab37-469473e8e931\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c167c0-6ccddcb77d-czh4t"
Aug 02 10:32:31.725013 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:31.724990    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-df2nz\" (UniqueName: \"kubernetes.io/projected/7a3c555e-1c47-44b0-ab37-469473e8e931-kube-api-access-df2nz\") pod \"maas-api-e2e-ipp-b-c167c0-6ccddcb77d-czh4t\" (UID: \"7a3c555e-1c47-44b0-ab37-469473e8e931\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c167c0-6ccddcb77d-czh4t"
Aug 02 10:32:31.952187 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:31.952153    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:32:32.057203 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:32.057165    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:32:32.155274 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:32.155220    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:32:32.216341 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:32.216260    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a3c555e-1c47-44b0-ab37-469473e8e931-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c167c0-6ccddcb77d-czh4t\" (UID: \"7a3c555e-1c47-44b0-ab37-469473e8e931\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c167c0-6ccddcb77d-czh4t"
Aug 02 10:32:32.218575 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:32.218552    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a3c555e-1c47-44b0-ab37-469473e8e931-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c167c0-6ccddcb77d-czh4t\" (UID: \"7a3c555e-1c47-44b0-ab37-469473e8e931\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c167c0-6ccddcb77d-czh4t"
Aug 02 10:32:32.416455 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:32.416424    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c167c0-6ccddcb77d-czh4t"
Aug 02 10:32:32.556584 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:32.556550    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c167c0-6ccddcb77d-czh4t"]
Aug 02 10:32:32.560554 ip-10-0-141-232 kubenswrapper[2564]: W0802 10:32:32.560529    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a3c555e_1c47_44b0_ab37_469473e8e931.slice/crio-5e2b43f7fbdafa5511328bd7c7e84bb940ceef6eb2534dadecf01c9c25c0928b WatchSource:0}: Error finding container 5e2b43f7fbdafa5511328bd7c7e84bb940ceef6eb2534dadecf01c9c25c0928b: Status 404 returned error can't find the container with id 5e2b43f7fbdafa5511328bd7c7e84bb940ceef6eb2534dadecf01c9c25c0928b
Aug 02 10:32:33.153969 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:33.153932    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c167c0-6ccddcb77d-czh4t" event={"ID":"7a3c555e-1c47-44b0-ab37-469473e8e931","Type":"ContainerStarted","Data":"7665d9bf6fcad5e4450eadf686e31bfd27294155e089fff8210fcfda60c1b2b2"}
Aug 02 10:32:33.154157 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:33.153985    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c167c0-6ccddcb77d-czh4t" event={"ID":"7a3c555e-1c47-44b0-ab37-469473e8e931","Type":"ContainerStarted","Data":"5e2b43f7fbdafa5511328bd7c7e84bb940ceef6eb2534dadecf01c9c25c0928b"}
Aug 02 10:32:33.154157 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:33.154054    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c167c0-6ccddcb77d-czh4t"
Aug 02 10:32:33.172520 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:33.172474    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c167c0-6ccddcb77d-czh4t" podStartSLOduration=1.881808369 podStartE2EDuration="2.172460029s" totalImagesPullingTime="290.65166ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:32:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:32:33.170164873 +0000 UTC m=+3576.884970319" watchObservedRunningTime="2026-08-02 10:32:33.172460029 +0000 UTC m=+3576.887265476"
Aug 02 10:32:39.164170 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:39.164142    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c167c0-6ccddcb77d-czh4t"
Aug 02 10:32:55.265528 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:32:55.265441    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:33:02.314169 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:02.314140    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-7b8zc_9eda18ed-b699-4efd-b283-47c0ac424ecb/payload-processing/0.log"
Aug 02 10:33:10.923038 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:10.923002    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:33:23.543842 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:23.543804    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:33:47.256399 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:47.256345    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-5c4717-d6dcf56b8-7nbm4_db34d767-89e9-4806-80df-7f88b724297f/payload-processing/0.log"
Aug 02 10:33:47.379386 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:47.379362    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-7b8zc_9eda18ed-b699-4efd-b283-47c0ac424ecb/payload-processing/0.log"
Aug 02 10:33:47.825398 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:47.825333    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:33:48.107370 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:48.107281    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:33:48.676191 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:48.676153    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:33:50.659698 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:50.659660    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:33:50.767178 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:50.767137    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:33:51.758846 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:51.758818    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-80d516-66cc4d594d-lhgds"]
Aug 02 10:33:51.762312 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:51.762295    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-80d516-66cc4d594d-lhgds"
Aug 02 10:33:51.764411 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:51.764388    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-80d516-serving-cert\""
Aug 02 10:33:51.773534 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:51.773513    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-80d516-66cc4d594d-lhgds"]
Aug 02 10:33:51.794840 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:51.794818    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:33:51.891480 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:51.891455    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7n4d\" (UniqueName: \"kubernetes.io/projected/4d681463-b676-4a8a-847c-30cf453b797b-kube-api-access-l7n4d\") pod \"maas-api-e2e-ipp-cleanup-80d516-66cc4d594d-lhgds\" (UID: \"4d681463-b676-4a8a-847c-30cf453b797b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-80d516-66cc4d594d-lhgds"
Aug 02 10:33:51.891637 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:51.891524    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d681463-b676-4a8a-847c-30cf453b797b-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-80d516-66cc4d594d-lhgds\" (UID: \"4d681463-b676-4a8a-847c-30cf453b797b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-80d516-66cc4d594d-lhgds"
Aug 02 10:33:51.891637 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:51.891585    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d681463-b676-4a8a-847c-30cf453b797b-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-80d516-66cc4d594d-lhgds\" (UID: \"4d681463-b676-4a8a-847c-30cf453b797b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-80d516-66cc4d594d-lhgds"
Aug 02 10:33:51.992508 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:51.992479    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d681463-b676-4a8a-847c-30cf453b797b-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-80d516-66cc4d594d-lhgds\" (UID: \"4d681463-b676-4a8a-847c-30cf453b797b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-80d516-66cc4d594d-lhgds"
Aug 02 10:33:51.992649 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:51.992518    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d681463-b676-4a8a-847c-30cf453b797b-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-80d516-66cc4d594d-lhgds\" (UID: \"4d681463-b676-4a8a-847c-30cf453b797b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-80d516-66cc4d594d-lhgds"
Aug 02 10:33:51.992649 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:51.992544    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7n4d\" (UniqueName: \"kubernetes.io/projected/4d681463-b676-4a8a-847c-30cf453b797b-kube-api-access-l7n4d\") pod \"maas-api-e2e-ipp-cleanup-80d516-66cc4d594d-lhgds\" (UID: \"4d681463-b676-4a8a-847c-30cf453b797b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-80d516-66cc4d594d-lhgds"
Aug 02 10:33:51.992649 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:33:51.992610    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-80d516-serving-cert: secret "maas-api-e2e-ipp-cleanup-80d516-serving-cert" not found
Aug 02 10:33:51.992779 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:33:51.992668    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d681463-b676-4a8a-847c-30cf453b797b-maas-api-tls podName:4d681463-b676-4a8a-847c-30cf453b797b nodeName:}" failed. No retries permitted until 2026-08-02 10:33:52.492652101 +0000 UTC m=+3656.207457527 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4d681463-b676-4a8a-847c-30cf453b797b-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-80d516-66cc4d594d-lhgds" (UID: "4d681463-b676-4a8a-847c-30cf453b797b") : secret "maas-api-e2e-ipp-cleanup-80d516-serving-cert" not found
Aug 02 10:33:51.993126 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:51.993110    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d681463-b676-4a8a-847c-30cf453b797b-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-80d516-66cc4d594d-lhgds\" (UID: \"4d681463-b676-4a8a-847c-30cf453b797b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-80d516-66cc4d594d-lhgds"
Aug 02 10:33:52.006652 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:52.006628    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7n4d\" (UniqueName: \"kubernetes.io/projected/4d681463-b676-4a8a-847c-30cf453b797b-kube-api-access-l7n4d\") pod \"maas-api-e2e-ipp-cleanup-80d516-66cc4d594d-lhgds\" (UID: \"4d681463-b676-4a8a-847c-30cf453b797b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-80d516-66cc4d594d-lhgds"
Aug 02 10:33:52.051444 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:52.051378    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:33:52.295357 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:52.295321    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:33:52.394248 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:52.394208    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:33:52.430702 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:52.430669    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-80d516-7f85697c9b-n46f8"]
Aug 02 10:33:52.434038 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:52.434022    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-80d516-7f85697c9b-n46f8"
Aug 02 10:33:52.437998 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:52.437979    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-80d516-dockercfg-wdqzw\""
Aug 02 10:33:52.438263 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:52.438228    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-80d516\""
Aug 02 10:33:52.456103 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:52.456074    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-80d516-7f85697c9b-n46f8"]
Aug 02 10:33:52.497469 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:52.497440    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t22x\" (UniqueName: \"kubernetes.io/projected/5c69aaa8-9034-476c-9c78-e7cb25624941-kube-api-access-8t22x\") pod \"payload-pre-processing-e2e-ipp-cleanup-80d516-7f85697c9b-n46f8\" (UID: \"5c69aaa8-9034-476c-9c78-e7cb25624941\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-80d516-7f85697c9b-n46f8"
Aug 02 10:33:52.497618 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:52.497482    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c69aaa8-9034-476c-9c78-e7cb25624941-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-80d516-7f85697c9b-n46f8\" (UID: \"5c69aaa8-9034-476c-9c78-e7cb25624941\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-80d516-7f85697c9b-n46f8"
Aug 02 10:33:52.497666 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:52.497645    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d681463-b676-4a8a-847c-30cf453b797b-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-80d516-66cc4d594d-lhgds\" (UID: \"4d681463-b676-4a8a-847c-30cf453b797b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-80d516-66cc4d594d-lhgds"
Aug 02 10:33:52.500061 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:52.500040    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d681463-b676-4a8a-847c-30cf453b797b-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-80d516-66cc4d594d-lhgds\" (UID: \"4d681463-b676-4a8a-847c-30cf453b797b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-80d516-66cc4d594d-lhgds"
Aug 02 10:33:52.598375 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:52.598341    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t22x\" (UniqueName: \"kubernetes.io/projected/5c69aaa8-9034-476c-9c78-e7cb25624941-kube-api-access-8t22x\") pod \"payload-pre-processing-e2e-ipp-cleanup-80d516-7f85697c9b-n46f8\" (UID: \"5c69aaa8-9034-476c-9c78-e7cb25624941\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-80d516-7f85697c9b-n46f8"
Aug 02 10:33:52.598547 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:52.598387    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c69aaa8-9034-476c-9c78-e7cb25624941-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-80d516-7f85697c9b-n46f8\" (UID: \"5c69aaa8-9034-476c-9c78-e7cb25624941\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-80d516-7f85697c9b-n46f8"
Aug 02 10:33:52.599031 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:52.599013    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c69aaa8-9034-476c-9c78-e7cb25624941-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-80d516-7f85697c9b-n46f8\" (UID: \"5c69aaa8-9034-476c-9c78-e7cb25624941\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-80d516-7f85697c9b-n46f8"
Aug 02 10:33:52.614013 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:52.613989    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t22x\" (UniqueName: \"kubernetes.io/projected/5c69aaa8-9034-476c-9c78-e7cb25624941-kube-api-access-8t22x\") pod \"payload-pre-processing-e2e-ipp-cleanup-80d516-7f85697c9b-n46f8\" (UID: \"5c69aaa8-9034-476c-9c78-e7cb25624941\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-80d516-7f85697c9b-n46f8"
Aug 02 10:33:52.673213 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:52.673134    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-80d516-66cc4d594d-lhgds"
Aug 02 10:33:52.744556 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:52.744527    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-80d516-7f85697c9b-n46f8"
Aug 02 10:33:52.815122 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:52.815099    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-80d516-66cc4d594d-lhgds"]
Aug 02 10:33:52.817059 ip-10-0-141-232 kubenswrapper[2564]: W0802 10:33:52.817029    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d681463_b676_4a8a_847c_30cf453b797b.slice/crio-f580b903379df8ea9f5eeb93ac516350bc2abebf396a353ff7b683d85c0094cf WatchSource:0}: Error finding container f580b903379df8ea9f5eeb93ac516350bc2abebf396a353ff7b683d85c0094cf: Status 404 returned error can't find the container with id f580b903379df8ea9f5eeb93ac516350bc2abebf396a353ff7b683d85c0094cf
Aug 02 10:33:52.890616 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:52.890586    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-80d516-7f85697c9b-n46f8"]
Aug 02 10:33:52.891108 ip-10-0-141-232 kubenswrapper[2564]: W0802 10:33:52.891080    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c69aaa8_9034_476c_9c78_e7cb25624941.slice/crio-5fee66557c40f26d16120c2bf1488ff28189523e4e5438e9e67ea9e598e7745e WatchSource:0}: Error finding container 5fee66557c40f26d16120c2bf1488ff28189523e4e5438e9e67ea9e598e7745e: Status 404 returned error can't find the container with id 5fee66557c40f26d16120c2bf1488ff28189523e4e5438e9e67ea9e598e7745e
Aug 02 10:33:53.502336 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:53.502224    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-80d516-66cc4d594d-lhgds" event={"ID":"4d681463-b676-4a8a-847c-30cf453b797b","Type":"ContainerStarted","Data":"7e81536bb618f26f2cece80f7cf3db46b066d2d639b0b2cd7dc9187f6db70ab5"}
Aug 02 10:33:53.502336 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:53.502298    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-80d516-66cc4d594d-lhgds" event={"ID":"4d681463-b676-4a8a-847c-30cf453b797b","Type":"ContainerStarted","Data":"f580b903379df8ea9f5eeb93ac516350bc2abebf396a353ff7b683d85c0094cf"}
Aug 02 10:33:53.502336 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:53.502324    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-80d516-66cc4d594d-lhgds"
Aug 02 10:33:53.503675 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:53.503656    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-80d516-7f85697c9b-n46f8" event={"ID":"5c69aaa8-9034-476c-9c78-e7cb25624941","Type":"ContainerStarted","Data":"55bad0fe2617a37cd70ca1754e26a70d3b1729f64672712ce8e4fed051c45d76"}
Aug 02 10:33:53.503783 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:53.503695    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-80d516-7f85697c9b-n46f8" event={"ID":"5c69aaa8-9034-476c-9c78-e7cb25624941","Type":"ContainerStarted","Data":"5fee66557c40f26d16120c2bf1488ff28189523e4e5438e9e67ea9e598e7745e"}
Aug 02 10:33:53.503783 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:53.503721    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-80d516-7f85697c9b-n46f8"
Aug 02 10:33:53.522375 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:53.522336    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-80d516-66cc4d594d-lhgds" podStartSLOduration=2.174378099 podStartE2EDuration="2.522323169s" totalImagesPullingTime="347.94507ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:33:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:33:53.520756947 +0000 UTC m=+3657.235562406" watchObservedRunningTime="2026-08-02 10:33:53.522323169 +0000 UTC m=+3657.237128617"
Aug 02 10:33:53.537245 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:53.537202    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-80d516-7f85697c9b-n46f8" podStartSLOduration=1.53718816 podStartE2EDuration="1.53718816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:33:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:33:53.536563204 +0000 UTC m=+3657.251368652" watchObservedRunningTime="2026-08-02 10:33:53.53718816 +0000 UTC m=+3657.251993608"
Aug 02 10:33:58.394754 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:58.394718    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:33:59.513522 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:33:59.513489    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-80d516-66cc4d594d-lhgds"
Aug 02 10:34:04.510551 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:04.510522    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-80d516-7f85697c9b-n46f8"
Aug 02 10:34:15.649287 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:15.649222    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:34:22.483725 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:22.483645    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-80d516-66cc4d594d-lhgds"]
Aug 02 10:34:22.484155 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:22.483903    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-80d516-66cc4d594d-lhgds" podUID="4d681463-b676-4a8a-847c-30cf453b797b" containerName="maas-api" containerID="cri-o://7e81536bb618f26f2cece80f7cf3db46b066d2d639b0b2cd7dc9187f6db70ab5" gracePeriod=30
Aug 02 10:34:22.518208 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:22.518177    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:34:22.555580 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:22.555543    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-80d516-7f85697c9b-n46f8"]
Aug 02 10:34:22.555824 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:22.555793    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-80d516-7f85697c9b-n46f8" podUID="5c69aaa8-9034-476c-9c78-e7cb25624941" containerName="payload-pre-processing" containerID="cri-o://55bad0fe2617a37cd70ca1754e26a70d3b1729f64672712ce8e4fed051c45d76" gracePeriod=30
Aug 02 10:34:22.616460 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:22.616426    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:34:22.634411 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:22.634386    2564 generic.go:356] "Generic (PLEG): container finished" podID="4d681463-b676-4a8a-847c-30cf453b797b" containerID="7e81536bb618f26f2cece80f7cf3db46b066d2d639b0b2cd7dc9187f6db70ab5" exitCode=0
Aug 02 10:34:22.634552 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:22.634466    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-80d516-66cc4d594d-lhgds" event={"ID":"4d681463-b676-4a8a-847c-30cf453b797b","Type":"ContainerDied","Data":"7e81536bb618f26f2cece80f7cf3db46b066d2d639b0b2cd7dc9187f6db70ab5"}
Aug 02 10:34:22.699373 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:22.699307    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:34:22.760091 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:22.760014    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:34:22.799809 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:22.799787    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-80d516-66cc4d594d-lhgds"
Aug 02 10:34:22.839829 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:22.839810    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-80d516-7f85697c9b-n46f8"
Aug 02 10:34:22.961341 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:22.961308    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d681463-b676-4a8a-847c-30cf453b797b-openshift-service-ca\") pod \"4d681463-b676-4a8a-847c-30cf453b797b\" (UID: \"4d681463-b676-4a8a-847c-30cf453b797b\") "
Aug 02 10:34:22.961533 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:22.961353    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8t22x\" (UniqueName: \"kubernetes.io/projected/5c69aaa8-9034-476c-9c78-e7cb25624941-kube-api-access-8t22x\") pod \"5c69aaa8-9034-476c-9c78-e7cb25624941\" (UID: \"5c69aaa8-9034-476c-9c78-e7cb25624941\") "
Aug 02 10:34:22.961533 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:22.961406    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c69aaa8-9034-476c-9c78-e7cb25624941-plugins-config-volume\") pod \"5c69aaa8-9034-476c-9c78-e7cb25624941\" (UID: \"5c69aaa8-9034-476c-9c78-e7cb25624941\") "
Aug 02 10:34:22.961533 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:22.961444    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7n4d\" (UniqueName: \"kubernetes.io/projected/4d681463-b676-4a8a-847c-30cf453b797b-kube-api-access-l7n4d\") pod \"4d681463-b676-4a8a-847c-30cf453b797b\" (UID: \"4d681463-b676-4a8a-847c-30cf453b797b\") "
Aug 02 10:34:22.961533 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:22.961490    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d681463-b676-4a8a-847c-30cf453b797b-maas-api-tls\") pod \"4d681463-b676-4a8a-847c-30cf453b797b\" (UID: \"4d681463-b676-4a8a-847c-30cf453b797b\") "
Aug 02 10:34:22.961778 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:22.961747    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d681463-b676-4a8a-847c-30cf453b797b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4d681463-b676-4a8a-847c-30cf453b797b" (UID: "4d681463-b676-4a8a-847c-30cf453b797b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 02 10:34:22.961863 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:22.961834    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c69aaa8-9034-476c-9c78-e7cb25624941-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5c69aaa8-9034-476c-9c78-e7cb25624941" (UID: "5c69aaa8-9034-476c-9c78-e7cb25624941"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 02 10:34:22.963592 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:22.963562    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d681463-b676-4a8a-847c-30cf453b797b-kube-api-access-l7n4d" (OuterVolumeSpecName: "kube-api-access-l7n4d") pod "4d681463-b676-4a8a-847c-30cf453b797b" (UID: "4d681463-b676-4a8a-847c-30cf453b797b"). InnerVolumeSpecName "kube-api-access-l7n4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 02 10:34:22.963703 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:22.963596    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d681463-b676-4a8a-847c-30cf453b797b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4d681463-b676-4a8a-847c-30cf453b797b" (UID: "4d681463-b676-4a8a-847c-30cf453b797b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 02 10:34:22.963703 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:22.963626    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c69aaa8-9034-476c-9c78-e7cb25624941-kube-api-access-8t22x" (OuterVolumeSpecName: "kube-api-access-8t22x") pod "5c69aaa8-9034-476c-9c78-e7cb25624941" (UID: "5c69aaa8-9034-476c-9c78-e7cb25624941"). InnerVolumeSpecName "kube-api-access-8t22x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 02 10:34:23.063216 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:23.063134    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d681463-b676-4a8a-847c-30cf453b797b-openshift-service-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:34:23.063216 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:23.063161    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t22x\" (UniqueName: \"kubernetes.io/projected/5c69aaa8-9034-476c-9c78-e7cb25624941-kube-api-access-8t22x\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:34:23.063216 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:23.063170    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c69aaa8-9034-476c-9c78-e7cb25624941-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:34:23.063216 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:23.063182    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7n4d\" (UniqueName: \"kubernetes.io/projected/4d681463-b676-4a8a-847c-30cf453b797b-kube-api-access-l7n4d\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:34:23.063216 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:23.063190    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d681463-b676-4a8a-847c-30cf453b797b-maas-api-tls\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:34:23.088715 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:23.088680    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:34:23.640041 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:23.639974    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-80d516-66cc4d594d-lhgds"
Aug 02 10:34:23.640041 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:23.640015    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-80d516-66cc4d594d-lhgds" event={"ID":"4d681463-b676-4a8a-847c-30cf453b797b","Type":"ContainerDied","Data":"f580b903379df8ea9f5eeb93ac516350bc2abebf396a353ff7b683d85c0094cf"}
Aug 02 10:34:23.640769 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:23.640082    2564 scope.go:117] "RemoveContainer" containerID="7e81536bb618f26f2cece80f7cf3db46b066d2d639b0b2cd7dc9187f6db70ab5"
Aug 02 10:34:23.641786 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:23.641576    2564 generic.go:356] "Generic (PLEG): container finished" podID="5c69aaa8-9034-476c-9c78-e7cb25624941" containerID="55bad0fe2617a37cd70ca1754e26a70d3b1729f64672712ce8e4fed051c45d76" exitCode=0
Aug 02 10:34:23.641786 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:23.641622    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-80d516-7f85697c9b-n46f8"
Aug 02 10:34:23.641786 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:23.641651    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-80d516-7f85697c9b-n46f8" event={"ID":"5c69aaa8-9034-476c-9c78-e7cb25624941","Type":"ContainerDied","Data":"55bad0fe2617a37cd70ca1754e26a70d3b1729f64672712ce8e4fed051c45d76"}
Aug 02 10:34:23.641786 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:23.641718    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-80d516-7f85697c9b-n46f8" event={"ID":"5c69aaa8-9034-476c-9c78-e7cb25624941","Type":"ContainerDied","Data":"5fee66557c40f26d16120c2bf1488ff28189523e4e5438e9e67ea9e598e7745e"}
Aug 02 10:34:23.650536 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:23.650517    2564 scope.go:117] "RemoveContainer" containerID="55bad0fe2617a37cd70ca1754e26a70d3b1729f64672712ce8e4fed051c45d76"
Aug 02 10:34:23.658147 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:23.658129    2564 scope.go:117] "RemoveContainer" containerID="55bad0fe2617a37cd70ca1754e26a70d3b1729f64672712ce8e4fed051c45d76"
Aug 02 10:34:23.658393 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:34:23.658377    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55bad0fe2617a37cd70ca1754e26a70d3b1729f64672712ce8e4fed051c45d76\": container with ID starting with 55bad0fe2617a37cd70ca1754e26a70d3b1729f64672712ce8e4fed051c45d76 not found: ID does not exist" containerID="55bad0fe2617a37cd70ca1754e26a70d3b1729f64672712ce8e4fed051c45d76"
Aug 02 10:34:23.658449 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:23.658399    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55bad0fe2617a37cd70ca1754e26a70d3b1729f64672712ce8e4fed051c45d76"} err="failed to get container status \"55bad0fe2617a37cd70ca1754e26a70d3b1729f64672712ce8e4fed051c45d76\": rpc error: code = NotFound desc = could not find container \"55bad0fe2617a37cd70ca1754e26a70d3b1729f64672712ce8e4fed051c45d76\": container with ID starting with 55bad0fe2617a37cd70ca1754e26a70d3b1729f64672712ce8e4fed051c45d76 not found: ID does not exist"
Aug 02 10:34:23.668549 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:23.668527    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-80d516-7f85697c9b-n46f8"]
Aug 02 10:34:23.676489 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:23.676469    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-80d516-7f85697c9b-n46f8"]
Aug 02 10:34:23.695636 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:23.695615    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-80d516-66cc4d594d-lhgds"]
Aug 02 10:34:23.700582 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:23.700563    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-80d516-66cc4d594d-lhgds"]
Aug 02 10:34:24.880323 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:24.880294    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d681463-b676-4a8a-847c-30cf453b797b" path="/var/lib/kubelet/pods/4d681463-b676-4a8a-847c-30cf453b797b/volumes"
Aug 02 10:34:24.880693 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:24.880642    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c69aaa8-9034-476c-9c78-e7cb25624941" path="/var/lib/kubelet/pods/5c69aaa8-9034-476c-9c78-e7cb25624941/volumes"
Aug 02 10:34:28.786993 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:28.786956    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:34:28.887353 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:28.887321    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:34:29.069364 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:29.069278    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:34:29.281885 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:29.281855    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:34:48.121328 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.121291    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c4717-c8c95bbbd-bvtlt"]
Aug 02 10:34:48.121957 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.121511    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c4717-c8c95bbbd-bvtlt" podUID="62d25586-5c2b-4399-b1f2-e64d695d4aa6" containerName="maas-api" containerID="cri-o://48c968604963d144ce9c44c79a54f0e1c95f2bea03b16fa19921a58f68cb2293" gracePeriod=30
Aug 02 10:34:48.153376 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.153347    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:34:48.169919 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.169894    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5c4717-d6dcf56b8-7nbm4"]
Aug 02 10:34:48.170123 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.170094    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c4717-d6dcf56b8-7nbm4" podUID="db34d767-89e9-4806-80df-7f88b724297f" containerName="payload-processing" containerID="cri-o://13c2775513e8858d38c61757435da728dfa1bc3b5544f1b1eb48cc2b13137704" gracePeriod=30
Aug 02 10:34:48.177919 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.177896    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5c4717-854ddd7664-nfp79"]
Aug 02 10:34:48.178092 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.178071    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c4717-854ddd7664-nfp79" podUID="54152ea7-799b-46d3-aa04-563e69cf9fb0" containerName="payload-pre-processing" containerID="cri-o://94d15a1a5a0d26f111a5f3da5ac1e291409a70029b9f76f53dc1e4bb872c25c1" gracePeriod=30
Aug 02 10:34:48.251694 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.251663    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:34:48.320644 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.320583    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:34:48.382443 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.382389    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:34:48.455959 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.455938    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c4717-c8c95bbbd-bvtlt"
Aug 02 10:34:48.483017 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.482997    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c4717-d6dcf56b8-7nbm4"
Aug 02 10:34:48.486228 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.486212    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c4717-854ddd7664-nfp79"
Aug 02 10:34:48.566094 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.566069    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db34d767-89e9-4806-80df-7f88b724297f-plugins-config-volume\") pod \"db34d767-89e9-4806-80df-7f88b724297f\" (UID: \"db34d767-89e9-4806-80df-7f88b724297f\") "
Aug 02 10:34:48.566300 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.566113    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzg4d\" (UniqueName: \"kubernetes.io/projected/62d25586-5c2b-4399-b1f2-e64d695d4aa6-kube-api-access-jzg4d\") pod \"62d25586-5c2b-4399-b1f2-e64d695d4aa6\" (UID: \"62d25586-5c2b-4399-b1f2-e64d695d4aa6\") "
Aug 02 10:34:48.566300 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.566132    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62d25586-5c2b-4399-b1f2-e64d695d4aa6-openshift-service-ca\") pod \"62d25586-5c2b-4399-b1f2-e64d695d4aa6\" (UID: \"62d25586-5c2b-4399-b1f2-e64d695d4aa6\") "
Aug 02 10:34:48.566300 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.566158    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62d25586-5c2b-4399-b1f2-e64d695d4aa6-maas-api-tls\") pod \"62d25586-5c2b-4399-b1f2-e64d695d4aa6\" (UID: \"62d25586-5c2b-4399-b1f2-e64d695d4aa6\") "
Aug 02 10:34:48.566300 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.566199    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdbw7\" (UniqueName: \"kubernetes.io/projected/db34d767-89e9-4806-80df-7f88b724297f-kube-api-access-qdbw7\") pod \"db34d767-89e9-4806-80df-7f88b724297f\" (UID: \"db34d767-89e9-4806-80df-7f88b724297f\") "
Aug 02 10:34:48.566300 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.566277    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lnzjk\" (UniqueName: \"kubernetes.io/projected/54152ea7-799b-46d3-aa04-563e69cf9fb0-kube-api-access-lnzjk\") pod \"54152ea7-799b-46d3-aa04-563e69cf9fb0\" (UID: \"54152ea7-799b-46d3-aa04-563e69cf9fb0\") "
Aug 02 10:34:48.566300 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.566298    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54152ea7-799b-46d3-aa04-563e69cf9fb0-plugins-config-volume\") pod \"54152ea7-799b-46d3-aa04-563e69cf9fb0\" (UID: \"54152ea7-799b-46d3-aa04-563e69cf9fb0\") "
Aug 02 10:34:48.566637 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.566523    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62d25586-5c2b-4399-b1f2-e64d695d4aa6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "62d25586-5c2b-4399-b1f2-e64d695d4aa6" (UID: "62d25586-5c2b-4399-b1f2-e64d695d4aa6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 02 10:34:48.566637 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.566573    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db34d767-89e9-4806-80df-7f88b724297f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "db34d767-89e9-4806-80df-7f88b724297f" (UID: "db34d767-89e9-4806-80df-7f88b724297f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 02 10:34:48.566802 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.566782    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54152ea7-799b-46d3-aa04-563e69cf9fb0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "54152ea7-799b-46d3-aa04-563e69cf9fb0" (UID: "54152ea7-799b-46d3-aa04-563e69cf9fb0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 02 10:34:48.568441 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.568418    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db34d767-89e9-4806-80df-7f88b724297f-kube-api-access-qdbw7" (OuterVolumeSpecName: "kube-api-access-qdbw7") pod "db34d767-89e9-4806-80df-7f88b724297f" (UID: "db34d767-89e9-4806-80df-7f88b724297f"). InnerVolumeSpecName "kube-api-access-qdbw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 02 10:34:48.568441 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.568429    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54152ea7-799b-46d3-aa04-563e69cf9fb0-kube-api-access-lnzjk" (OuterVolumeSpecName: "kube-api-access-lnzjk") pod "54152ea7-799b-46d3-aa04-563e69cf9fb0" (UID: "54152ea7-799b-46d3-aa04-563e69cf9fb0"). InnerVolumeSpecName "kube-api-access-lnzjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 02 10:34:48.568675 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.568656    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62d25586-5c2b-4399-b1f2-e64d695d4aa6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "62d25586-5c2b-4399-b1f2-e64d695d4aa6" (UID: "62d25586-5c2b-4399-b1f2-e64d695d4aa6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 02 10:34:48.568675 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.568665    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62d25586-5c2b-4399-b1f2-e64d695d4aa6-kube-api-access-jzg4d" (OuterVolumeSpecName: "kube-api-access-jzg4d") pod "62d25586-5c2b-4399-b1f2-e64d695d4aa6" (UID: "62d25586-5c2b-4399-b1f2-e64d695d4aa6"). InnerVolumeSpecName "kube-api-access-jzg4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 02 10:34:48.667811 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.667756    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzg4d\" (UniqueName: \"kubernetes.io/projected/62d25586-5c2b-4399-b1f2-e64d695d4aa6-kube-api-access-jzg4d\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:34:48.667811 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.667776    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62d25586-5c2b-4399-b1f2-e64d695d4aa6-openshift-service-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:34:48.667811 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.667786    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62d25586-5c2b-4399-b1f2-e64d695d4aa6-maas-api-tls\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:34:48.667811 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.667796    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdbw7\" (UniqueName: \"kubernetes.io/projected/db34d767-89e9-4806-80df-7f88b724297f-kube-api-access-qdbw7\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:34:48.667811 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.667806    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnzjk\" (UniqueName: \"kubernetes.io/projected/54152ea7-799b-46d3-aa04-563e69cf9fb0-kube-api-access-lnzjk\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:34:48.667811 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.667815    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54152ea7-799b-46d3-aa04-563e69cf9fb0-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:34:48.668058 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.667824    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db34d767-89e9-4806-80df-7f88b724297f-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:34:48.704447 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.704422    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:34:48.756182 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.756157    2564 generic.go:356] "Generic (PLEG): container finished" podID="62d25586-5c2b-4399-b1f2-e64d695d4aa6" containerID="48c968604963d144ce9c44c79a54f0e1c95f2bea03b16fa19921a58f68cb2293" exitCode=0
Aug 02 10:34:48.756296 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.756197    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c4717-c8c95bbbd-bvtlt"
Aug 02 10:34:48.756296 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.756258    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c4717-c8c95bbbd-bvtlt" event={"ID":"62d25586-5c2b-4399-b1f2-e64d695d4aa6","Type":"ContainerDied","Data":"48c968604963d144ce9c44c79a54f0e1c95f2bea03b16fa19921a58f68cb2293"}
Aug 02 10:34:48.756413 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.756303    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c4717-c8c95bbbd-bvtlt" event={"ID":"62d25586-5c2b-4399-b1f2-e64d695d4aa6","Type":"ContainerDied","Data":"a19e5b96f2f97daf7f4bfa089be39be262816466b2c9ce41e1139991f41891e0"}
Aug 02 10:34:48.756413 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.756327    2564 scope.go:117] "RemoveContainer" containerID="48c968604963d144ce9c44c79a54f0e1c95f2bea03b16fa19921a58f68cb2293"
Aug 02 10:34:48.757659 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.757640    2564 generic.go:356] "Generic (PLEG): container finished" podID="db34d767-89e9-4806-80df-7f88b724297f" containerID="13c2775513e8858d38c61757435da728dfa1bc3b5544f1b1eb48cc2b13137704" exitCode=0
Aug 02 10:34:48.757763 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.757683    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c4717-d6dcf56b8-7nbm4"
Aug 02 10:34:48.757763 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.757718    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c4717-d6dcf56b8-7nbm4" event={"ID":"db34d767-89e9-4806-80df-7f88b724297f","Type":"ContainerDied","Data":"13c2775513e8858d38c61757435da728dfa1bc3b5544f1b1eb48cc2b13137704"}
Aug 02 10:34:48.757763 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.757753    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c4717-d6dcf56b8-7nbm4" event={"ID":"db34d767-89e9-4806-80df-7f88b724297f","Type":"ContainerDied","Data":"f8ea84460fb928328accc0d46fb99616c1bfb692001fa0c84652cd5d2ac9c00f"}
Aug 02 10:34:48.759155 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.759116    2564 generic.go:356] "Generic (PLEG): container finished" podID="54152ea7-799b-46d3-aa04-563e69cf9fb0" containerID="94d15a1a5a0d26f111a5f3da5ac1e291409a70029b9f76f53dc1e4bb872c25c1" exitCode=0
Aug 02 10:34:48.759300 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.759184    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c4717-854ddd7664-nfp79"
Aug 02 10:34:48.759300 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.759214    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c4717-854ddd7664-nfp79" event={"ID":"54152ea7-799b-46d3-aa04-563e69cf9fb0","Type":"ContainerDied","Data":"94d15a1a5a0d26f111a5f3da5ac1e291409a70029b9f76f53dc1e4bb872c25c1"}
Aug 02 10:34:48.759300 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.759262    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5c4717-854ddd7664-nfp79" event={"ID":"54152ea7-799b-46d3-aa04-563e69cf9fb0","Type":"ContainerDied","Data":"b92f6d35e2f71970e74e5c659a7348595df554f87748f53fe5287993e941b62f"}
Aug 02 10:34:48.766380 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.766362    2564 scope.go:117] "RemoveContainer" containerID="48c968604963d144ce9c44c79a54f0e1c95f2bea03b16fa19921a58f68cb2293"
Aug 02 10:34:48.766696 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:34:48.766669    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48c968604963d144ce9c44c79a54f0e1c95f2bea03b16fa19921a58f68cb2293\": container with ID starting with 48c968604963d144ce9c44c79a54f0e1c95f2bea03b16fa19921a58f68cb2293 not found: ID does not exist" containerID="48c968604963d144ce9c44c79a54f0e1c95f2bea03b16fa19921a58f68cb2293"
Aug 02 10:34:48.766765 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.766708    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48c968604963d144ce9c44c79a54f0e1c95f2bea03b16fa19921a58f68cb2293"} err="failed to get container status \"48c968604963d144ce9c44c79a54f0e1c95f2bea03b16fa19921a58f68cb2293\": rpc error: code = NotFound desc = could not find container \"48c968604963d144ce9c44c79a54f0e1c95f2bea03b16fa19921a58f68cb2293\": container with ID starting with 48c968604963d144ce9c44c79a54f0e1c95f2bea03b16fa19921a58f68cb2293 not found: ID does not exist"
Aug 02 10:34:48.766765 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.766732    2564 scope.go:117] "RemoveContainer" containerID="13c2775513e8858d38c61757435da728dfa1bc3b5544f1b1eb48cc2b13137704"
Aug 02 10:34:48.774873 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.774859    2564 scope.go:117] "RemoveContainer" containerID="13c2775513e8858d38c61757435da728dfa1bc3b5544f1b1eb48cc2b13137704"
Aug 02 10:34:48.775107 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:34:48.775086    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13c2775513e8858d38c61757435da728dfa1bc3b5544f1b1eb48cc2b13137704\": container with ID starting with 13c2775513e8858d38c61757435da728dfa1bc3b5544f1b1eb48cc2b13137704 not found: ID does not exist" containerID="13c2775513e8858d38c61757435da728dfa1bc3b5544f1b1eb48cc2b13137704"
Aug 02 10:34:48.775190 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.775115    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13c2775513e8858d38c61757435da728dfa1bc3b5544f1b1eb48cc2b13137704"} err="failed to get container status \"13c2775513e8858d38c61757435da728dfa1bc3b5544f1b1eb48cc2b13137704\": rpc error: code = NotFound desc = could not find container \"13c2775513e8858d38c61757435da728dfa1bc3b5544f1b1eb48cc2b13137704\": container with ID starting with 13c2775513e8858d38c61757435da728dfa1bc3b5544f1b1eb48cc2b13137704 not found: ID does not exist"
Aug 02 10:34:48.775190 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.775137    2564 scope.go:117] "RemoveContainer" containerID="94d15a1a5a0d26f111a5f3da5ac1e291409a70029b9f76f53dc1e4bb872c25c1"
Aug 02 10:34:48.780628 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.780608    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c4717-c8c95bbbd-bvtlt"]
Aug 02 10:34:48.782286 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.782266    2564 scope.go:117] "RemoveContainer" containerID="94d15a1a5a0d26f111a5f3da5ac1e291409a70029b9f76f53dc1e4bb872c25c1"
Aug 02 10:34:48.782675 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:34:48.782650    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94d15a1a5a0d26f111a5f3da5ac1e291409a70029b9f76f53dc1e4bb872c25c1\": container with ID starting with 94d15a1a5a0d26f111a5f3da5ac1e291409a70029b9f76f53dc1e4bb872c25c1 not found: ID does not exist" containerID="94d15a1a5a0d26f111a5f3da5ac1e291409a70029b9f76f53dc1e4bb872c25c1"
Aug 02 10:34:48.782781 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.782754    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94d15a1a5a0d26f111a5f3da5ac1e291409a70029b9f76f53dc1e4bb872c25c1"} err="failed to get container status \"94d15a1a5a0d26f111a5f3da5ac1e291409a70029b9f76f53dc1e4bb872c25c1\": rpc error: code = NotFound desc = could not find container \"94d15a1a5a0d26f111a5f3da5ac1e291409a70029b9f76f53dc1e4bb872c25c1\": container with ID starting with 94d15a1a5a0d26f111a5f3da5ac1e291409a70029b9f76f53dc1e4bb872c25c1 not found: ID does not exist"
Aug 02 10:34:48.784507 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.784488    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c4717-c8c95bbbd-bvtlt"]
Aug 02 10:34:48.793952 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.793933    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5c4717-d6dcf56b8-7nbm4"]
Aug 02 10:34:48.799767 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.799743    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5c4717-d6dcf56b8-7nbm4"]
Aug 02 10:34:48.810103 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.810086    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5c4717-854ddd7664-nfp79"]
Aug 02 10:34:48.813449 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.813431    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5c4717-854ddd7664-nfp79"]
Aug 02 10:34:48.880575 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.880552    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54152ea7-799b-46d3-aa04-563e69cf9fb0" path="/var/lib/kubelet/pods/54152ea7-799b-46d3-aa04-563e69cf9fb0/volumes"
Aug 02 10:34:48.880911 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.880899    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62d25586-5c2b-4399-b1f2-e64d695d4aa6" path="/var/lib/kubelet/pods/62d25586-5c2b-4399-b1f2-e64d695d4aa6/volumes"
Aug 02 10:34:48.881195 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:48.881184    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db34d767-89e9-4806-80df-7f88b724297f" path="/var/lib/kubelet/pods/db34d767-89e9-4806-80df-7f88b724297f/volumes"
Aug 02 10:34:53.737207 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:34:53.737164    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:35:22.946504 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:35:22.946466    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:35:30.404903 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:35:30.404868    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:35:30.701260 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:35:30.701154    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:35:36.319264 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:35:36.319212    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:35:36.424770 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:35:36.424732    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:35:36.607299 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:35:36.607196    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:35:36.871968 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:35:36.871880    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:35:47.397558 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:35:47.397524    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c167c0-6ccddcb77d-czh4t"]
Aug 02 10:35:47.398025 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:35:47.397772    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c167c0-6ccddcb77d-czh4t" podUID="7a3c555e-1c47-44b0-ab37-469473e8e931" containerName="maas-api" containerID="cri-o://7665d9bf6fcad5e4450eadf686e31bfd27294155e089fff8210fcfda60c1b2b2" gracePeriod=30
Aug 02 10:35:47.424921 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:35:47.424888    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:35:47.509777 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:35:47.509743    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:35:47.578164 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:35:47.578002    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:35:47.650500 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:35:47.650437    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:35:47.652434 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:35:47.652417    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c167c0-6ccddcb77d-czh4t"
Aug 02 10:35:47.720920 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:35:47.720894    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a3c555e-1c47-44b0-ab37-469473e8e931-openshift-service-ca\") pod \"7a3c555e-1c47-44b0-ab37-469473e8e931\" (UID: \"7a3c555e-1c47-44b0-ab37-469473e8e931\") "
Aug 02 10:35:47.721067 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:35:47.720944    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-df2nz\" (UniqueName: \"kubernetes.io/projected/7a3c555e-1c47-44b0-ab37-469473e8e931-kube-api-access-df2nz\") pod \"7a3c555e-1c47-44b0-ab37-469473e8e931\" (UID: \"7a3c555e-1c47-44b0-ab37-469473e8e931\") "
Aug 02 10:35:47.721067 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:35:47.720969    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a3c555e-1c47-44b0-ab37-469473e8e931-maas-api-tls\") pod \"7a3c555e-1c47-44b0-ab37-469473e8e931\" (UID: \"7a3c555e-1c47-44b0-ab37-469473e8e931\") "
Aug 02 10:35:47.721327 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:35:47.721303    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a3c555e-1c47-44b0-ab37-469473e8e931-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7a3c555e-1c47-44b0-ab37-469473e8e931" (UID: "7a3c555e-1c47-44b0-ab37-469473e8e931"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 02 10:35:47.723083 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:35:47.723060    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a3c555e-1c47-44b0-ab37-469473e8e931-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7a3c555e-1c47-44b0-ab37-469473e8e931" (UID: "7a3c555e-1c47-44b0-ab37-469473e8e931"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 02 10:35:47.723083 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:35:47.723071    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a3c555e-1c47-44b0-ab37-469473e8e931-kube-api-access-df2nz" (OuterVolumeSpecName: "kube-api-access-df2nz") pod "7a3c555e-1c47-44b0-ab37-469473e8e931" (UID: "7a3c555e-1c47-44b0-ab37-469473e8e931"). InnerVolumeSpecName "kube-api-access-df2nz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 02 10:35:47.821594 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:35:47.821563    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a3c555e-1c47-44b0-ab37-469473e8e931-openshift-service-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:35:47.821594 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:35:47.821591    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df2nz\" (UniqueName: \"kubernetes.io/projected/7a3c555e-1c47-44b0-ab37-469473e8e931-kube-api-access-df2nz\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:35:47.821594 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:35:47.821601    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a3c555e-1c47-44b0-ab37-469473e8e931-maas-api-tls\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:35:48.010003 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:35:48.009925    2564 generic.go:356] "Generic (PLEG): container finished" podID="7a3c555e-1c47-44b0-ab37-469473e8e931" containerID="7665d9bf6fcad5e4450eadf686e31bfd27294155e089fff8210fcfda60c1b2b2" exitCode=0
Aug 02 10:35:48.010003 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:35:48.009960    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c167c0-6ccddcb77d-czh4t" event={"ID":"7a3c555e-1c47-44b0-ab37-469473e8e931","Type":"ContainerDied","Data":"7665d9bf6fcad5e4450eadf686e31bfd27294155e089fff8210fcfda60c1b2b2"}
Aug 02 10:35:48.010003 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:35:48.009964    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c167c0-6ccddcb77d-czh4t"
Aug 02 10:35:48.010003 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:35:48.009996    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c167c0-6ccddcb77d-czh4t" event={"ID":"7a3c555e-1c47-44b0-ab37-469473e8e931","Type":"ContainerDied","Data":"5e2b43f7fbdafa5511328bd7c7e84bb940ceef6eb2534dadecf01c9c25c0928b"}
Aug 02 10:35:48.010308 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:35:48.010018    2564 scope.go:117] "RemoveContainer" containerID="7665d9bf6fcad5e4450eadf686e31bfd27294155e089fff8210fcfda60c1b2b2"
Aug 02 10:35:48.017697 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:35:48.017668    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:35:48.018828 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:35:48.018812    2564 scope.go:117] "RemoveContainer" containerID="7665d9bf6fcad5e4450eadf686e31bfd27294155e089fff8210fcfda60c1b2b2"
Aug 02 10:35:48.019075 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:35:48.019053    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7665d9bf6fcad5e4450eadf686e31bfd27294155e089fff8210fcfda60c1b2b2\": container with ID starting with 7665d9bf6fcad5e4450eadf686e31bfd27294155e089fff8210fcfda60c1b2b2 not found: ID does not exist" containerID="7665d9bf6fcad5e4450eadf686e31bfd27294155e089fff8210fcfda60c1b2b2"
Aug 02 10:35:48.019176 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:35:48.019081    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7665d9bf6fcad5e4450eadf686e31bfd27294155e089fff8210fcfda60c1b2b2"} err="failed to get container status \"7665d9bf6fcad5e4450eadf686e31bfd27294155e089fff8210fcfda60c1b2b2\": rpc error: code = NotFound desc = could not find container \"7665d9bf6fcad5e4450eadf686e31bfd27294155e089fff8210fcfda60c1b2b2\": container with ID starting with 7665d9bf6fcad5e4450eadf686e31bfd27294155e089fff8210fcfda60c1b2b2 not found: ID does not exist"
Aug 02 10:35:48.030773 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:35:48.030753    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c167c0-6ccddcb77d-czh4t"]
Aug 02 10:35:48.035225 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:35:48.035204    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c167c0-6ccddcb77d-czh4t"]
Aug 02 10:35:48.882285 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:35:48.882254    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a3c555e-1c47-44b0-ab37-469473e8e931" path="/var/lib/kubelet/pods/7a3c555e-1c47-44b0-ab37-469473e8e931/volumes"
Aug 02 10:35:54.096699 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:35:54.096645    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:35:54.196696 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:35:54.196664    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:35:54.375730 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:35:54.375700    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:35:54.624630 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:35:54.624594    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:36:05.186417 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:05.186382    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e9225c-76cf7db755-rc2bb"]
Aug 02 10:36:05.186907 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:05.186644    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e9225c-76cf7db755-rc2bb" podUID="0f970ab0-becb-42e7-bdf0-e04a8267c531" containerName="maas-api" containerID="cri-o://7d5123b7d75e1d26e20dc9cb221b82347a1cbf2fea10dd0cfb8b4d180ce19a45" gracePeriod=30
Aug 02 10:36:05.222655 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:05.222617    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:36:05.234960 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:05.234888    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-e9225c-85b98c998-48zlf"]
Aug 02 10:36:05.235385 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:05.235356    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-e9225c-85b98c998-48zlf" podUID="b0a74cc0-3669-4745-b2b9-e3af72f2671f" containerName="payload-processing" containerID="cri-o://84b82df9a78d886fa0d4aede6b7318e7b64a325a1d3ebf87a7eeb47ff3fc7262" gracePeriod=30
Aug 02 10:36:05.240352 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:05.240330    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-e9225c-7d9f9d5fb-pmc6f"]
Aug 02 10:36:05.240565 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:05.240541    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-e9225c-7d9f9d5fb-pmc6f" podUID="71a2ede0-fd50-4c40-8076-047501f50391" containerName="payload-pre-processing" containerID="cri-o://c221aa89fa3b24952d6b4178544e65506076c508b8d79bfae981bdf57a26bf68" gracePeriod=30
Aug 02 10:36:05.322751 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:05.322711    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:36:05.390487 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:05.390442    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:36:05.526836 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:05.526813    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-e9225c-85b98c998-48zlf"
Aug 02 10:36:05.529015 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:05.528995    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-e9225c-7d9f9d5fb-pmc6f"
Aug 02 10:36:05.533826 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:05.533810    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e9225c-76cf7db755-rc2bb"
Aug 02 10:36:05.667740 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:05.667705    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71a2ede0-fd50-4c40-8076-047501f50391-plugins-config-volume\") pod \"71a2ede0-fd50-4c40-8076-047501f50391\" (UID: \"71a2ede0-fd50-4c40-8076-047501f50391\") "
Aug 02 10:36:05.667740 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:05.667744    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rl7cc\" (UniqueName: \"kubernetes.io/projected/b0a74cc0-3669-4745-b2b9-e3af72f2671f-kube-api-access-rl7cc\") pod \"b0a74cc0-3669-4745-b2b9-e3af72f2671f\" (UID: \"b0a74cc0-3669-4745-b2b9-e3af72f2671f\") "
Aug 02 10:36:05.667969 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:05.667775    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f970ab0-becb-42e7-bdf0-e04a8267c531-openshift-service-ca\") pod \"0f970ab0-becb-42e7-bdf0-e04a8267c531\" (UID: \"0f970ab0-becb-42e7-bdf0-e04a8267c531\") "
Aug 02 10:36:05.667969 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:05.667817    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0a74cc0-3669-4745-b2b9-e3af72f2671f-plugins-config-volume\") pod \"b0a74cc0-3669-4745-b2b9-e3af72f2671f\" (UID: \"b0a74cc0-3669-4745-b2b9-e3af72f2671f\") "
Aug 02 10:36:05.667969 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:05.667837    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4h8s\" (UniqueName: \"kubernetes.io/projected/0f970ab0-becb-42e7-bdf0-e04a8267c531-kube-api-access-s4h8s\") pod \"0f970ab0-becb-42e7-bdf0-e04a8267c531\" (UID: \"0f970ab0-becb-42e7-bdf0-e04a8267c531\") "
Aug 02 10:36:05.667969 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:05.667890    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f970ab0-becb-42e7-bdf0-e04a8267c531-maas-api-tls\") pod \"0f970ab0-becb-42e7-bdf0-e04a8267c531\" (UID: \"0f970ab0-becb-42e7-bdf0-e04a8267c531\") "
Aug 02 10:36:05.667969 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:05.667944    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgmpv\" (UniqueName: \"kubernetes.io/projected/71a2ede0-fd50-4c40-8076-047501f50391-kube-api-access-bgmpv\") pod \"71a2ede0-fd50-4c40-8076-047501f50391\" (UID: \"71a2ede0-fd50-4c40-8076-047501f50391\") "
Aug 02 10:36:05.668266 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:05.668122    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71a2ede0-fd50-4c40-8076-047501f50391-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "71a2ede0-fd50-4c40-8076-047501f50391" (UID: "71a2ede0-fd50-4c40-8076-047501f50391"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 02 10:36:05.668266 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:05.668135    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f970ab0-becb-42e7-bdf0-e04a8267c531-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0f970ab0-becb-42e7-bdf0-e04a8267c531" (UID: "0f970ab0-becb-42e7-bdf0-e04a8267c531"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 02 10:36:05.668266 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:05.668209    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0a74cc0-3669-4745-b2b9-e3af72f2671f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b0a74cc0-3669-4745-b2b9-e3af72f2671f" (UID: "b0a74cc0-3669-4745-b2b9-e3af72f2671f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 02 10:36:05.668529 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:05.668496    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0a74cc0-3669-4745-b2b9-e3af72f2671f-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:36:05.668529 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:05.668520    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71a2ede0-fd50-4c40-8076-047501f50391-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:36:05.668635 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:05.668536    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f970ab0-becb-42e7-bdf0-e04a8267c531-openshift-service-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:36:05.669986 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:05.669956    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71a2ede0-fd50-4c40-8076-047501f50391-kube-api-access-bgmpv" (OuterVolumeSpecName: "kube-api-access-bgmpv") pod "71a2ede0-fd50-4c40-8076-047501f50391" (UID: "71a2ede0-fd50-4c40-8076-047501f50391"). InnerVolumeSpecName "kube-api-access-bgmpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 02 10:36:05.670075 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:05.670001    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f970ab0-becb-42e7-bdf0-e04a8267c531-kube-api-access-s4h8s" (OuterVolumeSpecName: "kube-api-access-s4h8s") pod "0f970ab0-becb-42e7-bdf0-e04a8267c531" (UID: "0f970ab0-becb-42e7-bdf0-e04a8267c531"). InnerVolumeSpecName "kube-api-access-s4h8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 02 10:36:05.670338 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:05.670316    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f970ab0-becb-42e7-bdf0-e04a8267c531-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0f970ab0-becb-42e7-bdf0-e04a8267c531" (UID: "0f970ab0-becb-42e7-bdf0-e04a8267c531"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 02 10:36:05.670390 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:05.670325    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0a74cc0-3669-4745-b2b9-e3af72f2671f-kube-api-access-rl7cc" (OuterVolumeSpecName: "kube-api-access-rl7cc") pod "b0a74cc0-3669-4745-b2b9-e3af72f2671f" (UID: "b0a74cc0-3669-4745-b2b9-e3af72f2671f"). InnerVolumeSpecName "kube-api-access-rl7cc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 02 10:36:05.768911 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:05.768831    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4h8s\" (UniqueName: \"kubernetes.io/projected/0f970ab0-becb-42e7-bdf0-e04a8267c531-kube-api-access-s4h8s\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:36:05.768911 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:05.768856    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f970ab0-becb-42e7-bdf0-e04a8267c531-maas-api-tls\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:36:05.768911 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:05.768866    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgmpv\" (UniqueName: \"kubernetes.io/projected/71a2ede0-fd50-4c40-8076-047501f50391-kube-api-access-bgmpv\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:36:05.768911 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:05.768875    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl7cc\" (UniqueName: \"kubernetes.io/projected/b0a74cc0-3669-4745-b2b9-e3af72f2671f-kube-api-access-rl7cc\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:36:05.788301 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:05.788250    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:36:06.088369 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:06.088295    2564 generic.go:356] "Generic (PLEG): container finished" podID="b0a74cc0-3669-4745-b2b9-e3af72f2671f" containerID="84b82df9a78d886fa0d4aede6b7318e7b64a325a1d3ebf87a7eeb47ff3fc7262" exitCode=0
Aug 02 10:36:06.088369 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:06.088341    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-e9225c-85b98c998-48zlf"
Aug 02 10:36:06.088559 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:06.088379    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-e9225c-85b98c998-48zlf" event={"ID":"b0a74cc0-3669-4745-b2b9-e3af72f2671f","Type":"ContainerDied","Data":"84b82df9a78d886fa0d4aede6b7318e7b64a325a1d3ebf87a7eeb47ff3fc7262"}
Aug 02 10:36:06.088559 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:06.088429    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-e9225c-85b98c998-48zlf" event={"ID":"b0a74cc0-3669-4745-b2b9-e3af72f2671f","Type":"ContainerDied","Data":"cce5918404f182f0d3cdfd78a7f0884ff7d4f7c8857a83d63f0e77da1eb3112f"}
Aug 02 10:36:06.088559 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:06.088449    2564 scope.go:117] "RemoveContainer" containerID="84b82df9a78d886fa0d4aede6b7318e7b64a325a1d3ebf87a7eeb47ff3fc7262"
Aug 02 10:36:06.089725 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:06.089707    2564 generic.go:356] "Generic (PLEG): container finished" podID="71a2ede0-fd50-4c40-8076-047501f50391" containerID="c221aa89fa3b24952d6b4178544e65506076c508b8d79bfae981bdf57a26bf68" exitCode=0
Aug 02 10:36:06.089818 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:06.089744    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-e9225c-7d9f9d5fb-pmc6f" event={"ID":"71a2ede0-fd50-4c40-8076-047501f50391","Type":"ContainerDied","Data":"c221aa89fa3b24952d6b4178544e65506076c508b8d79bfae981bdf57a26bf68"}
Aug 02 10:36:06.089818 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:06.089796    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-e9225c-7d9f9d5fb-pmc6f" event={"ID":"71a2ede0-fd50-4c40-8076-047501f50391","Type":"ContainerDied","Data":"5435e62ef333c40dac07942653d963ece0628ab0a6aa6ac7cf94df9a25ef7613"}
Aug 02 10:36:06.089818 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:06.089755    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-e9225c-7d9f9d5fb-pmc6f"
Aug 02 10:36:06.091157 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:06.091132    2564 generic.go:356] "Generic (PLEG): container finished" podID="0f970ab0-becb-42e7-bdf0-e04a8267c531" containerID="7d5123b7d75e1d26e20dc9cb221b82347a1cbf2fea10dd0cfb8b4d180ce19a45" exitCode=0
Aug 02 10:36:06.091294 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:06.091186    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e9225c-76cf7db755-rc2bb"
Aug 02 10:36:06.091294 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:06.091201    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e9225c-76cf7db755-rc2bb" event={"ID":"0f970ab0-becb-42e7-bdf0-e04a8267c531","Type":"ContainerDied","Data":"7d5123b7d75e1d26e20dc9cb221b82347a1cbf2fea10dd0cfb8b4d180ce19a45"}
Aug 02 10:36:06.091294 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:06.091252    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e9225c-76cf7db755-rc2bb" event={"ID":"0f970ab0-becb-42e7-bdf0-e04a8267c531","Type":"ContainerDied","Data":"fef3dff8f2cab841e36524a8072386d5cdc3653350b3cb2b0a8188752939ef80"}
Aug 02 10:36:06.097291 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:06.097223    2564 scope.go:117] "RemoveContainer" containerID="84b82df9a78d886fa0d4aede6b7318e7b64a325a1d3ebf87a7eeb47ff3fc7262"
Aug 02 10:36:06.097511 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:36:06.097490    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84b82df9a78d886fa0d4aede6b7318e7b64a325a1d3ebf87a7eeb47ff3fc7262\": container with ID starting with 84b82df9a78d886fa0d4aede6b7318e7b64a325a1d3ebf87a7eeb47ff3fc7262 not found: ID does not exist" containerID="84b82df9a78d886fa0d4aede6b7318e7b64a325a1d3ebf87a7eeb47ff3fc7262"
Aug 02 10:36:06.097595 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:06.097516    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84b82df9a78d886fa0d4aede6b7318e7b64a325a1d3ebf87a7eeb47ff3fc7262"} err="failed to get container status \"84b82df9a78d886fa0d4aede6b7318e7b64a325a1d3ebf87a7eeb47ff3fc7262\": rpc error: code = NotFound desc = could not find container \"84b82df9a78d886fa0d4aede6b7318e7b64a325a1d3ebf87a7eeb47ff3fc7262\": container with ID starting with 84b82df9a78d886fa0d4aede6b7318e7b64a325a1d3ebf87a7eeb47ff3fc7262 not found: ID does not exist"
Aug 02 10:36:06.097595 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:06.097537    2564 scope.go:117] "RemoveContainer" containerID="c221aa89fa3b24952d6b4178544e65506076c508b8d79bfae981bdf57a26bf68"
Aug 02 10:36:06.106447 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:06.106431    2564 scope.go:117] "RemoveContainer" containerID="c221aa89fa3b24952d6b4178544e65506076c508b8d79bfae981bdf57a26bf68"
Aug 02 10:36:06.106693 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:36:06.106676    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c221aa89fa3b24952d6b4178544e65506076c508b8d79bfae981bdf57a26bf68\": container with ID starting with c221aa89fa3b24952d6b4178544e65506076c508b8d79bfae981bdf57a26bf68 not found: ID does not exist" containerID="c221aa89fa3b24952d6b4178544e65506076c508b8d79bfae981bdf57a26bf68"
Aug 02 10:36:06.106772 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:06.106703    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c221aa89fa3b24952d6b4178544e65506076c508b8d79bfae981bdf57a26bf68"} err="failed to get container status \"c221aa89fa3b24952d6b4178544e65506076c508b8d79bfae981bdf57a26bf68\": rpc error: code = NotFound desc = could not find container \"c221aa89fa3b24952d6b4178544e65506076c508b8d79bfae981bdf57a26bf68\": container with ID starting with c221aa89fa3b24952d6b4178544e65506076c508b8d79bfae981bdf57a26bf68 not found: ID does not exist"
Aug 02 10:36:06.106772 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:06.106723    2564 scope.go:117] "RemoveContainer" containerID="7d5123b7d75e1d26e20dc9cb221b82347a1cbf2fea10dd0cfb8b4d180ce19a45"
Aug 02 10:36:06.116181 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:06.115383    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-e9225c-85b98c998-48zlf"]
Aug 02 10:36:06.116181 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:06.115531    2564 scope.go:117] "RemoveContainer" containerID="7d5123b7d75e1d26e20dc9cb221b82347a1cbf2fea10dd0cfb8b4d180ce19a45"
Aug 02 10:36:06.116181 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:36:06.115893    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d5123b7d75e1d26e20dc9cb221b82347a1cbf2fea10dd0cfb8b4d180ce19a45\": container with ID starting with 7d5123b7d75e1d26e20dc9cb221b82347a1cbf2fea10dd0cfb8b4d180ce19a45 not found: ID does not exist" containerID="7d5123b7d75e1d26e20dc9cb221b82347a1cbf2fea10dd0cfb8b4d180ce19a45"
Aug 02 10:36:06.116181 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:06.116017    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d5123b7d75e1d26e20dc9cb221b82347a1cbf2fea10dd0cfb8b4d180ce19a45"} err="failed to get container status \"7d5123b7d75e1d26e20dc9cb221b82347a1cbf2fea10dd0cfb8b4d180ce19a45\": rpc error: code = NotFound desc = could not find container \"7d5123b7d75e1d26e20dc9cb221b82347a1cbf2fea10dd0cfb8b4d180ce19a45\": container with ID starting with 7d5123b7d75e1d26e20dc9cb221b82347a1cbf2fea10dd0cfb8b4d180ce19a45 not found: ID does not exist"
Aug 02 10:36:06.119064 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:06.119045    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-e9225c-85b98c998-48zlf"]
Aug 02 10:36:06.130521 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:06.130496    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e9225c-76cf7db755-rc2bb"]
Aug 02 10:36:06.136132 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:06.136112    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e9225c-76cf7db755-rc2bb"]
Aug 02 10:36:06.146185 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:06.146166    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-e9225c-7d9f9d5fb-pmc6f"]
Aug 02 10:36:06.150034 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:06.150014    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-e9225c-7d9f9d5fb-pmc6f"]
Aug 02 10:36:06.880092 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:06.880059    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f970ab0-becb-42e7-bdf0-e04a8267c531" path="/var/lib/kubelet/pods/0f970ab0-becb-42e7-bdf0-e04a8267c531/volumes"
Aug 02 10:36:06.880493 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:06.880418    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71a2ede0-fd50-4c40-8076-047501f50391" path="/var/lib/kubelet/pods/71a2ede0-fd50-4c40-8076-047501f50391/volumes"
Aug 02 10:36:06.880724 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:06.880711    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0a74cc0-3669-4745-b2b9-e3af72f2671f" path="/var/lib/kubelet/pods/b0a74cc0-3669-4745-b2b9-e3af72f2671f/volumes"
Aug 02 10:36:11.823457 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:11.823422    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:36:11.925174 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:11.925140    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:36:12.080777 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:12.080690    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:36:12.317297 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:12.317261    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:36:22.863519 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:22.863487    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-0c6483-bc76ffcb6-fvz7z"]
Aug 02 10:36:22.864178 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:22.863710    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0c6483-bc76ffcb6-fvz7z" podUID="14dd43af-5b06-41c4-9f85-bd3e4eae7f5b" containerName="maas-api" containerID="cri-o://579ebdcbab598b9502387e664aeaeb40de0bc841a1ff7a22eecda5b58330b614" gracePeriod=30
Aug 02 10:36:22.897553 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:22.897522    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:36:22.922901 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:22.922877    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-0c6483-59bf4c67c4-mcv6f"]
Aug 02 10:36:22.927554 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:22.927530    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-0c6483-dfc8557fb-wtg6n"]
Aug 02 10:36:22.927759 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:22.927733    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-0c6483-dfc8557fb-wtg6n" podUID="5284a7de-f4d1-4f6a-9f6d-94e8b6926527" containerName="payload-pre-processing" containerID="cri-o://29a593efb17af90016cfcaf4b6c706c6a25856c961de8330cf68210ea477ced2" gracePeriod=30
Aug 02 10:36:22.997461 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:36:22.997430    2564 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-shared-b-0c6483: configmap "payload-processing-plugins-e2e-shared-b-0c6483" not found
Aug 02 10:36:22.997612 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:36:22.997512    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6bbc1b38-00d1-4004-b341-1b7bb0c8e2b6-plugins-config-volume podName:6bbc1b38-00d1-4004-b341-1b7bb0c8e2b6 nodeName:}" failed. No retries permitted until 2026-08-02 10:36:23.497494924 +0000 UTC m=+3807.212300351 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/6bbc1b38-00d1-4004-b341-1b7bb0c8e2b6-plugins-config-volume") pod "payload-processing-e2e-shared-b-0c6483-59bf4c67c4-mcv6f" (UID: "6bbc1b38-00d1-4004-b341-1b7bb0c8e2b6") : configmap "payload-processing-plugins-e2e-shared-b-0c6483" not found
Aug 02 10:36:22.998005 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:22.997980    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:36:23.090768 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:23.090712    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:36:23.151152 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:23.151124    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:36:23.164715 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:23.164687    2564 generic.go:356] "Generic (PLEG): container finished" podID="14dd43af-5b06-41c4-9f85-bd3e4eae7f5b" containerID="579ebdcbab598b9502387e664aeaeb40de0bc841a1ff7a22eecda5b58330b614" exitCode=0
Aug 02 10:36:23.164857 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:23.164761    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0c6483-bc76ffcb6-fvz7z" event={"ID":"14dd43af-5b06-41c4-9f85-bd3e4eae7f5b","Type":"ContainerDied","Data":"579ebdcbab598b9502387e664aeaeb40de0bc841a1ff7a22eecda5b58330b614"}
Aug 02 10:36:23.166492 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:23.166472    2564 generic.go:356] "Generic (PLEG): container finished" podID="5284a7de-f4d1-4f6a-9f6d-94e8b6926527" containerID="29a593efb17af90016cfcaf4b6c706c6a25856c961de8330cf68210ea477ced2" exitCode=0
Aug 02 10:36:23.166708 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:23.166545    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-0c6483-dfc8557fb-wtg6n" event={"ID":"5284a7de-f4d1-4f6a-9f6d-94e8b6926527","Type":"ContainerDied","Data":"29a593efb17af90016cfcaf4b6c706c6a25856c961de8330cf68210ea477ced2"}
Aug 02 10:36:23.166708 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:23.166632    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-0c6483-59bf4c67c4-mcv6f" podUID="6bbc1b38-00d1-4004-b341-1b7bb0c8e2b6" containerName="payload-processing" containerID="cri-o://557d6540a0dc82f2ae292943bd2939d0124737a00e4889abf3e90cd364adf7d3" gracePeriod=30
Aug 02 10:36:23.246462 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:23.246436    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0c6483-bc76ffcb6-fvz7z"
Aug 02 10:36:23.278263 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:23.278244    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-0c6483-dfc8557fb-wtg6n"
Aug 02 10:36:23.398390 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:23.398359    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-0c6483-59bf4c67c4-mcv6f"
Aug 02 10:36:23.400400 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:23.400382    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14dd43af-5b06-41c4-9f85-bd3e4eae7f5b-maas-api-tls\") pod \"14dd43af-5b06-41c4-9f85-bd3e4eae7f5b\" (UID: \"14dd43af-5b06-41c4-9f85-bd3e4eae7f5b\") "
Aug 02 10:36:23.400494 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:23.400417    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14dd43af-5b06-41c4-9f85-bd3e4eae7f5b-openshift-service-ca\") pod \"14dd43af-5b06-41c4-9f85-bd3e4eae7f5b\" (UID: \"14dd43af-5b06-41c4-9f85-bd3e4eae7f5b\") "
Aug 02 10:36:23.400494 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:23.400434    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4lzmt\" (UniqueName: \"kubernetes.io/projected/5284a7de-f4d1-4f6a-9f6d-94e8b6926527-kube-api-access-4lzmt\") pod \"5284a7de-f4d1-4f6a-9f6d-94e8b6926527\" (UID: \"5284a7de-f4d1-4f6a-9f6d-94e8b6926527\") "
Aug 02 10:36:23.400494 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:23.400455    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqxvb\" (UniqueName: \"kubernetes.io/projected/14dd43af-5b06-41c4-9f85-bd3e4eae7f5b-kube-api-access-qqxvb\") pod \"14dd43af-5b06-41c4-9f85-bd3e4eae7f5b\" (UID: \"14dd43af-5b06-41c4-9f85-bd3e4eae7f5b\") "
Aug 02 10:36:23.400609 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:23.400492    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5284a7de-f4d1-4f6a-9f6d-94e8b6926527-plugins-config-volume\") pod \"5284a7de-f4d1-4f6a-9f6d-94e8b6926527\" (UID: \"5284a7de-f4d1-4f6a-9f6d-94e8b6926527\") "
Aug 02 10:36:23.400807 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:23.400779    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14dd43af-5b06-41c4-9f85-bd3e4eae7f5b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "14dd43af-5b06-41c4-9f85-bd3e4eae7f5b" (UID: "14dd43af-5b06-41c4-9f85-bd3e4eae7f5b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 02 10:36:23.400907 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:23.400875    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5284a7de-f4d1-4f6a-9f6d-94e8b6926527-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5284a7de-f4d1-4f6a-9f6d-94e8b6926527" (UID: "5284a7de-f4d1-4f6a-9f6d-94e8b6926527"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 02 10:36:23.402504 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:23.402452    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14dd43af-5b06-41c4-9f85-bd3e4eae7f5b-kube-api-access-qqxvb" (OuterVolumeSpecName: "kube-api-access-qqxvb") pod "14dd43af-5b06-41c4-9f85-bd3e4eae7f5b" (UID: "14dd43af-5b06-41c4-9f85-bd3e4eae7f5b"). InnerVolumeSpecName "kube-api-access-qqxvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 02 10:36:23.402585 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:23.402494    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5284a7de-f4d1-4f6a-9f6d-94e8b6926527-kube-api-access-4lzmt" (OuterVolumeSpecName: "kube-api-access-4lzmt") pod "5284a7de-f4d1-4f6a-9f6d-94e8b6926527" (UID: "5284a7de-f4d1-4f6a-9f6d-94e8b6926527"). InnerVolumeSpecName "kube-api-access-4lzmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 02 10:36:23.402585 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:23.402548    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14dd43af-5b06-41c4-9f85-bd3e4eae7f5b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "14dd43af-5b06-41c4-9f85-bd3e4eae7f5b" (UID: "14dd43af-5b06-41c4-9f85-bd3e4eae7f5b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 02 10:36:23.458570 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:23.458538    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:36:23.501339 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:23.501316    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-msbvm\" (UniqueName: \"kubernetes.io/projected/6bbc1b38-00d1-4004-b341-1b7bb0c8e2b6-kube-api-access-msbvm\") pod \"6bbc1b38-00d1-4004-b341-1b7bb0c8e2b6\" (UID: \"6bbc1b38-00d1-4004-b341-1b7bb0c8e2b6\") "
Aug 02 10:36:23.501429 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:23.501359    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bbc1b38-00d1-4004-b341-1b7bb0c8e2b6-plugins-config-volume\") pod \"6bbc1b38-00d1-4004-b341-1b7bb0c8e2b6\" (UID: \"6bbc1b38-00d1-4004-b341-1b7bb0c8e2b6\") "
Aug 02 10:36:23.501523 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:23.501513    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14dd43af-5b06-41c4-9f85-bd3e4eae7f5b-maas-api-tls\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:36:23.501559 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:23.501525    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14dd43af-5b06-41c4-9f85-bd3e4eae7f5b-openshift-service-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:36:23.501559 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:23.501536    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lzmt\" (UniqueName: \"kubernetes.io/projected/5284a7de-f4d1-4f6a-9f6d-94e8b6926527-kube-api-access-4lzmt\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:36:23.501559 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:23.501546    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqxvb\" (UniqueName: \"kubernetes.io/projected/14dd43af-5b06-41c4-9f85-bd3e4eae7f5b-kube-api-access-qqxvb\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:36:23.501559 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:23.501555    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5284a7de-f4d1-4f6a-9f6d-94e8b6926527-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:36:23.501724 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:23.501703    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bbc1b38-00d1-4004-b341-1b7bb0c8e2b6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6bbc1b38-00d1-4004-b341-1b7bb0c8e2b6" (UID: "6bbc1b38-00d1-4004-b341-1b7bb0c8e2b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 02 10:36:23.503213 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:23.503195    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bbc1b38-00d1-4004-b341-1b7bb0c8e2b6-kube-api-access-msbvm" (OuterVolumeSpecName: "kube-api-access-msbvm") pod "6bbc1b38-00d1-4004-b341-1b7bb0c8e2b6" (UID: "6bbc1b38-00d1-4004-b341-1b7bb0c8e2b6"). InnerVolumeSpecName "kube-api-access-msbvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 02 10:36:23.602719 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:23.602685    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msbvm\" (UniqueName: \"kubernetes.io/projected/6bbc1b38-00d1-4004-b341-1b7bb0c8e2b6-kube-api-access-msbvm\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:36:23.602719 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:23.602717    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bbc1b38-00d1-4004-b341-1b7bb0c8e2b6-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Aug 02 10:36:24.171756 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:24.171669    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0c6483-bc76ffcb6-fvz7z"
Aug 02 10:36:24.171756 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:24.171681    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0c6483-bc76ffcb6-fvz7z" event={"ID":"14dd43af-5b06-41c4-9f85-bd3e4eae7f5b","Type":"ContainerDied","Data":"9c8e5edeb87131d8ffeb3da9ad3bd7cd18e22c82b89402dc4203e0391dc2aa6b"}
Aug 02 10:36:24.171756 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:24.171746    2564 scope.go:117] "RemoveContainer" containerID="579ebdcbab598b9502387e664aeaeb40de0bc841a1ff7a22eecda5b58330b614"
Aug 02 10:36:24.173071 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:24.173049    2564 generic.go:356] "Generic (PLEG): container finished" podID="6bbc1b38-00d1-4004-b341-1b7bb0c8e2b6" containerID="557d6540a0dc82f2ae292943bd2939d0124737a00e4889abf3e90cd364adf7d3" exitCode=0
Aug 02 10:36:24.173149 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:24.173090    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-0c6483-59bf4c67c4-mcv6f"
Aug 02 10:36:24.173149 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:24.173108    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-0c6483-59bf4c67c4-mcv6f" event={"ID":"6bbc1b38-00d1-4004-b341-1b7bb0c8e2b6","Type":"ContainerDied","Data":"557d6540a0dc82f2ae292943bd2939d0124737a00e4889abf3e90cd364adf7d3"}
Aug 02 10:36:24.173284 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:24.173161    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-0c6483-59bf4c67c4-mcv6f" event={"ID":"6bbc1b38-00d1-4004-b341-1b7bb0c8e2b6","Type":"ContainerDied","Data":"d0fb4c1de81e51eb3289008acad5f6c7e5c8f81142d767449efd922ae1f407db"}
Aug 02 10:36:24.174421 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:24.174402    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-0c6483-dfc8557fb-wtg6n" event={"ID":"5284a7de-f4d1-4f6a-9f6d-94e8b6926527","Type":"ContainerDied","Data":"c17542634f9708e16598d620aabbd33c5358a6c20f11aee7815dd4b3b675b59f"}
Aug 02 10:36:24.174522 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:24.174442    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-0c6483-dfc8557fb-wtg6n"
Aug 02 10:36:24.182200 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:24.182164    2564 scope.go:117] "RemoveContainer" containerID="557d6540a0dc82f2ae292943bd2939d0124737a00e4889abf3e90cd364adf7d3"
Aug 02 10:36:24.190022 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:24.190006    2564 scope.go:117] "RemoveContainer" containerID="557d6540a0dc82f2ae292943bd2939d0124737a00e4889abf3e90cd364adf7d3"
Aug 02 10:36:24.190275 ip-10-0-141-232 kubenswrapper[2564]: E0802 10:36:24.190258    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"557d6540a0dc82f2ae292943bd2939d0124737a00e4889abf3e90cd364adf7d3\": container with ID starting with 557d6540a0dc82f2ae292943bd2939d0124737a00e4889abf3e90cd364adf7d3 not found: ID does not exist" containerID="557d6540a0dc82f2ae292943bd2939d0124737a00e4889abf3e90cd364adf7d3"
Aug 02 10:36:24.190325 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:24.190282    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"557d6540a0dc82f2ae292943bd2939d0124737a00e4889abf3e90cd364adf7d3"} err="failed to get container status \"557d6540a0dc82f2ae292943bd2939d0124737a00e4889abf3e90cd364adf7d3\": rpc error: code = NotFound desc = could not find container \"557d6540a0dc82f2ae292943bd2939d0124737a00e4889abf3e90cd364adf7d3\": container with ID starting with 557d6540a0dc82f2ae292943bd2939d0124737a00e4889abf3e90cd364adf7d3 not found: ID does not exist"
Aug 02 10:36:24.190325 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:24.190299    2564 scope.go:117] "RemoveContainer" containerID="29a593efb17af90016cfcaf4b6c706c6a25856c961de8330cf68210ea477ced2"
Aug 02 10:36:24.198481 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:24.198463    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-0c6483-59bf4c67c4-mcv6f"]
Aug 02 10:36:24.201771 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:24.201751    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-0c6483-59bf4c67c4-mcv6f"]
Aug 02 10:36:24.210902 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:24.210882    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-0c6483-bc76ffcb6-fvz7z"]
Aug 02 10:36:24.212970 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:24.212952    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-0c6483-bc76ffcb6-fvz7z"]
Aug 02 10:36:24.222539 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:24.222519    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-0c6483-dfc8557fb-wtg6n"]
Aug 02 10:36:24.224534 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:24.224513    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-0c6483-dfc8557fb-wtg6n"]
Aug 02 10:36:24.881088 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:24.881056    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14dd43af-5b06-41c4-9f85-bd3e4eae7f5b" path="/var/lib/kubelet/pods/14dd43af-5b06-41c4-9f85-bd3e4eae7f5b/volumes"
Aug 02 10:36:24.881434 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:24.881420    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5284a7de-f4d1-4f6a-9f6d-94e8b6926527" path="/var/lib/kubelet/pods/5284a7de-f4d1-4f6a-9f6d-94e8b6926527/volumes"
Aug 02 10:36:24.881727 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:24.881715    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bbc1b38-00d1-4004-b341-1b7bb0c8e2b6" path="/var/lib/kubelet/pods/6bbc1b38-00d1-4004-b341-1b7bb0c8e2b6/volumes"
Aug 02 10:36:29.594245 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:29.594188    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:36:29.656610 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:29.656573    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:36:29.865607 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:29.865524    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:36:30.093812 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:30.093777    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:36:32.988456 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:32.988421    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:36:33.052118 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:33.052089    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:36:36.680542 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:36.680511    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5694fb54d4-6z7v6_91e8b861-99fe-4311-9db5-ab8763266d31/copy-manifests/0.log"
Aug 02 10:36:36.700743 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:36.700718    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5694fb54d4-6z7v6_91e8b861-99fe-4311-9db5-ab8763266d31/manager/0.log"
Aug 02 10:36:37.291410 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:37.291367    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-856cd5f99f-rv47z_c674e553-e960-4aa7-8ccf-ef3da49ffa4f/maas-api/0.log"
Aug 02 10:36:37.410136 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:37.410096    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-2jcgx_e2233455-e64a-41d2-a92d-f884f674d629/postgres/0.log"
Aug 02 10:36:39.394526 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:39.394480    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-cbzmt_84d87887-e1dd-4a84-807d-6f551f7cbe8a/discovery/0.log"
Aug 02 10:36:39.868217 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:39.868140    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-hfw8f_d41871d5-cf2c-4676-8d98-96060bef7b10/payload-pre-processing/0.log"
Aug 02 10:36:39.989166 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:39.989132    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-7b8zc_9eda18ed-b699-4efd-b283-47c0ac424ecb/payload-processing/0.log"
Aug 02 10:36:40.462425 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:40.462395    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-b4629_e7e35e27-82ae-4072-b8e8-fe60a8a08bfc/main/0.log"
Aug 02 10:36:40.800835 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:40.800761    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flp55n_95023ac0-e83b-4b4c-9db9-de898589f429/main/0.log"
Aug 02 10:36:46.568246 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:46.568198    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:36:46.673748 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:46.673705    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log" limit=1024
Aug 02 10:36:49.562359 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:49.562329    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-cs5vh_e6944ae2-aaed-4a4a-9c46-dda99772fa54/global-pull-secret-syncer/0.log"
Aug 02 10:36:49.660891 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:49.660859    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-fhzz9_3f97d5c9-1a25-43c5-8050-9c619be890e4/konnectivity-agent/0.log"
Aug 02 10:36:49.801835 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:49.801806    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-232.ec2.internal_ce69cf0400f5bcb61dc1156a46d82c8f/haproxy/0.log"
Aug 02 10:36:57.503956 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:57.503861    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-88xrg_b8b72c7d-38e1-4478-ab62-465e91cf5434/node-exporter/0.log"
Aug 02 10:36:57.526588 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:57.526559    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-88xrg_b8b72c7d-38e1-4478-ab62-465e91cf5434/kube-rbac-proxy/0.log"
Aug 02 10:36:57.548800 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:57.548765    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-88xrg_b8b72c7d-38e1-4478-ab62-465e91cf5434/init-textfile/0.log"
Aug 02 10:36:59.810596 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.810565    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4kwt9/perf-node-gather-daemonset-j25x6"]
Aug 02 10:36:59.810979 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.810854    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0a74cc0-3669-4745-b2b9-e3af72f2671f" containerName="payload-processing"
Aug 02 10:36:59.810979 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.810866    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0a74cc0-3669-4745-b2b9-e3af72f2671f" containerName="payload-processing"
Aug 02 10:36:59.810979 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.810875    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14dd43af-5b06-41c4-9f85-bd3e4eae7f5b" containerName="maas-api"
Aug 02 10:36:59.810979 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.810880    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="14dd43af-5b06-41c4-9f85-bd3e4eae7f5b" containerName="maas-api"
Aug 02 10:36:59.810979 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.810890    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5284a7de-f4d1-4f6a-9f6d-94e8b6926527" containerName="payload-pre-processing"
Aug 02 10:36:59.810979 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.810896    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5284a7de-f4d1-4f6a-9f6d-94e8b6926527" containerName="payload-pre-processing"
Aug 02 10:36:59.810979 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.810904    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a3c555e-1c47-44b0-ab37-469473e8e931" containerName="maas-api"
Aug 02 10:36:59.810979 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.810910    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a3c555e-1c47-44b0-ab37-469473e8e931" containerName="maas-api"
Aug 02 10:36:59.810979 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.810916    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c69aaa8-9034-476c-9c78-e7cb25624941" containerName="payload-pre-processing"
Aug 02 10:36:59.810979 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.810922    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c69aaa8-9034-476c-9c78-e7cb25624941" containerName="payload-pre-processing"
Aug 02 10:36:59.810979 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.810930    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62d25586-5c2b-4399-b1f2-e64d695d4aa6" containerName="maas-api"
Aug 02 10:36:59.810979 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.810936    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="62d25586-5c2b-4399-b1f2-e64d695d4aa6" containerName="maas-api"
Aug 02 10:36:59.810979 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.810943    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71a2ede0-fd50-4c40-8076-047501f50391" containerName="payload-pre-processing"
Aug 02 10:36:59.810979 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.810948    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="71a2ede0-fd50-4c40-8076-047501f50391" containerName="payload-pre-processing"
Aug 02 10:36:59.810979 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.810955    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d681463-b676-4a8a-847c-30cf453b797b" containerName="maas-api"
Aug 02 10:36:59.810979 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.810960    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d681463-b676-4a8a-847c-30cf453b797b" containerName="maas-api"
Aug 02 10:36:59.810979 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.810967    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db34d767-89e9-4806-80df-7f88b724297f" containerName="payload-processing"
Aug 02 10:36:59.810979 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.810971    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="db34d767-89e9-4806-80df-7f88b724297f" containerName="payload-processing"
Aug 02 10:36:59.810979 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.810978    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f970ab0-becb-42e7-bdf0-e04a8267c531" containerName="maas-api"
Aug 02 10:36:59.810979 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.810983    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f970ab0-becb-42e7-bdf0-e04a8267c531" containerName="maas-api"
Aug 02 10:36:59.810979 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.810988    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bbc1b38-00d1-4004-b341-1b7bb0c8e2b6" containerName="payload-processing"
Aug 02 10:36:59.810979 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.810993    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bbc1b38-00d1-4004-b341-1b7bb0c8e2b6" containerName="payload-processing"
Aug 02 10:36:59.811740 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.811000    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54152ea7-799b-46d3-aa04-563e69cf9fb0" containerName="payload-pre-processing"
Aug 02 10:36:59.811740 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.811005    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="54152ea7-799b-46d3-aa04-563e69cf9fb0" containerName="payload-pre-processing"
Aug 02 10:36:59.811740 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.811059    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c69aaa8-9034-476c-9c78-e7cb25624941" containerName="payload-pre-processing"
Aug 02 10:36:59.811740 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.811066    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bbc1b38-00d1-4004-b341-1b7bb0c8e2b6" containerName="payload-processing"
Aug 02 10:36:59.811740 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.811073    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="db34d767-89e9-4806-80df-7f88b724297f" containerName="payload-processing"
Aug 02 10:36:59.811740 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.811079    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0a74cc0-3669-4745-b2b9-e3af72f2671f" containerName="payload-processing"
Aug 02 10:36:59.811740 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.811084    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="71a2ede0-fd50-4c40-8076-047501f50391" containerName="payload-pre-processing"
Aug 02 10:36:59.811740 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.811090    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f970ab0-becb-42e7-bdf0-e04a8267c531" containerName="maas-api"
Aug 02 10:36:59.811740 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.811095    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="14dd43af-5b06-41c4-9f85-bd3e4eae7f5b" containerName="maas-api"
Aug 02 10:36:59.811740 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.811100    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5284a7de-f4d1-4f6a-9f6d-94e8b6926527" containerName="payload-pre-processing"
Aug 02 10:36:59.811740 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.811105    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="62d25586-5c2b-4399-b1f2-e64d695d4aa6" containerName="maas-api"
Aug 02 10:36:59.811740 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.811110    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d681463-b676-4a8a-847c-30cf453b797b" containerName="maas-api"
Aug 02 10:36:59.811740 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.811115    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a3c555e-1c47-44b0-ab37-469473e8e931" containerName="maas-api"
Aug 02 10:36:59.811740 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.811119    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="54152ea7-799b-46d3-aa04-563e69cf9fb0" containerName="payload-pre-processing"
Aug 02 10:36:59.815441 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.815424    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4kwt9/perf-node-gather-daemonset-j25x6"
Aug 02 10:36:59.817588 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.817567    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4kwt9\"/\"openshift-service-ca.crt\""
Aug 02 10:36:59.817676 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.817613    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4kwt9\"/\"default-dockercfg-97nlp\""
Aug 02 10:36:59.818296 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.818280    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4kwt9\"/\"kube-root-ca.crt\""
Aug 02 10:36:59.822372 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.822070    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4kwt9/perf-node-gather-daemonset-j25x6"]
Aug 02 10:36:59.862186 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.862154    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3393e95d-2795-45cd-8fb6-7edc0e81720d-sys\") pod \"perf-node-gather-daemonset-j25x6\" (UID: \"3393e95d-2795-45cd-8fb6-7edc0e81720d\") " pod="openshift-must-gather-4kwt9/perf-node-gather-daemonset-j25x6"
Aug 02 10:36:59.862388 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.862218    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3393e95d-2795-45cd-8fb6-7edc0e81720d-proc\") pod \"perf-node-gather-daemonset-j25x6\" (UID: \"3393e95d-2795-45cd-8fb6-7edc0e81720d\") " pod="openshift-must-gather-4kwt9/perf-node-gather-daemonset-j25x6"
Aug 02 10:36:59.862388 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.862259    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3393e95d-2795-45cd-8fb6-7edc0e81720d-lib-modules\") pod \"perf-node-gather-daemonset-j25x6\" (UID: \"3393e95d-2795-45cd-8fb6-7edc0e81720d\") " pod="openshift-must-gather-4kwt9/perf-node-gather-daemonset-j25x6"
Aug 02 10:36:59.862388 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.862299    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3393e95d-2795-45cd-8fb6-7edc0e81720d-podres\") pod \"perf-node-gather-daemonset-j25x6\" (UID: \"3393e95d-2795-45cd-8fb6-7edc0e81720d\") " pod="openshift-must-gather-4kwt9/perf-node-gather-daemonset-j25x6"
Aug 02 10:36:59.862388 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.862351    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgmlv\" (UniqueName: \"kubernetes.io/projected/3393e95d-2795-45cd-8fb6-7edc0e81720d-kube-api-access-bgmlv\") pod \"perf-node-gather-daemonset-j25x6\" (UID: \"3393e95d-2795-45cd-8fb6-7edc0e81720d\") " pod="openshift-must-gather-4kwt9/perf-node-gather-daemonset-j25x6"
Aug 02 10:36:59.962752 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.962716    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3393e95d-2795-45cd-8fb6-7edc0e81720d-podres\") pod \"perf-node-gather-daemonset-j25x6\" (UID: \"3393e95d-2795-45cd-8fb6-7edc0e81720d\") " pod="openshift-must-gather-4kwt9/perf-node-gather-daemonset-j25x6"
Aug 02 10:36:59.962752 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.962753    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgmlv\" (UniqueName: \"kubernetes.io/projected/3393e95d-2795-45cd-8fb6-7edc0e81720d-kube-api-access-bgmlv\") pod \"perf-node-gather-daemonset-j25x6\" (UID: \"3393e95d-2795-45cd-8fb6-7edc0e81720d\") " pod="openshift-must-gather-4kwt9/perf-node-gather-daemonset-j25x6"
Aug 02 10:36:59.963072 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.962886    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3393e95d-2795-45cd-8fb6-7edc0e81720d-podres\") pod \"perf-node-gather-daemonset-j25x6\" (UID: \"3393e95d-2795-45cd-8fb6-7edc0e81720d\") " pod="openshift-must-gather-4kwt9/perf-node-gather-daemonset-j25x6"
Aug 02 10:36:59.963072 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.962885    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3393e95d-2795-45cd-8fb6-7edc0e81720d-sys\") pod \"perf-node-gather-daemonset-j25x6\" (UID: \"3393e95d-2795-45cd-8fb6-7edc0e81720d\") " pod="openshift-must-gather-4kwt9/perf-node-gather-daemonset-j25x6"
Aug 02 10:36:59.963072 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.962941    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3393e95d-2795-45cd-8fb6-7edc0e81720d-sys\") pod \"perf-node-gather-daemonset-j25x6\" (UID: \"3393e95d-2795-45cd-8fb6-7edc0e81720d\") " pod="openshift-must-gather-4kwt9/perf-node-gather-daemonset-j25x6"
Aug 02 10:36:59.963072 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.962962    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3393e95d-2795-45cd-8fb6-7edc0e81720d-proc\") pod \"perf-node-gather-daemonset-j25x6\" (UID: \"3393e95d-2795-45cd-8fb6-7edc0e81720d\") " pod="openshift-must-gather-4kwt9/perf-node-gather-daemonset-j25x6"
Aug 02 10:36:59.963072 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.962981    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3393e95d-2795-45cd-8fb6-7edc0e81720d-lib-modules\") pod \"perf-node-gather-daemonset-j25x6\" (UID: \"3393e95d-2795-45cd-8fb6-7edc0e81720d\") " pod="openshift-must-gather-4kwt9/perf-node-gather-daemonset-j25x6"
Aug 02 10:36:59.963260 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.963071    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3393e95d-2795-45cd-8fb6-7edc0e81720d-proc\") pod \"perf-node-gather-daemonset-j25x6\" (UID: \"3393e95d-2795-45cd-8fb6-7edc0e81720d\") " pod="openshift-must-gather-4kwt9/perf-node-gather-daemonset-j25x6"
Aug 02 10:36:59.963260 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.963088    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3393e95d-2795-45cd-8fb6-7edc0e81720d-lib-modules\") pod \"perf-node-gather-daemonset-j25x6\" (UID: \"3393e95d-2795-45cd-8fb6-7edc0e81720d\") " pod="openshift-must-gather-4kwt9/perf-node-gather-daemonset-j25x6"
Aug 02 10:36:59.970482 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:36:59.970454    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgmlv\" (UniqueName: \"kubernetes.io/projected/3393e95d-2795-45cd-8fb6-7edc0e81720d-kube-api-access-bgmlv\") pod \"perf-node-gather-daemonset-j25x6\" (UID: \"3393e95d-2795-45cd-8fb6-7edc0e81720d\") " pod="openshift-must-gather-4kwt9/perf-node-gather-daemonset-j25x6"
Aug 02 10:37:00.125772 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:37:00.125745    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4kwt9/perf-node-gather-daemonset-j25x6"
Aug 02 10:37:00.255700 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:37:00.255653    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4kwt9/perf-node-gather-daemonset-j25x6"]
Aug 02 10:37:00.261832 ip-10-0-141-232 kubenswrapper[2564]: W0802 10:37:00.261806    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3393e95d_2795_45cd_8fb6_7edc0e81720d.slice/crio-8596809f55dee0b1d7609365970930ae61f15318a2a0c0ed09e379cce8f8691c WatchSource:0}: Error finding container 8596809f55dee0b1d7609365970930ae61f15318a2a0c0ed09e379cce8f8691c: Status 404 returned error can't find the container with id 8596809f55dee0b1d7609365970930ae61f15318a2a0c0ed09e379cce8f8691c
Aug 02 10:37:00.330826 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:37:00.330801    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4kwt9/perf-node-gather-daemonset-j25x6" event={"ID":"3393e95d-2795-45cd-8fb6-7edc0e81720d","Type":"ContainerStarted","Data":"8596809f55dee0b1d7609365970930ae61f15318a2a0c0ed09e379cce8f8691c"}
Aug 02 10:37:01.338535 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:37:01.338501    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4kwt9/perf-node-gather-daemonset-j25x6" event={"ID":"3393e95d-2795-45cd-8fb6-7edc0e81720d","Type":"ContainerStarted","Data":"f2ca5eb685371cc50067515e3043b080108b5ab53089788f9cc4e1c89eb4e600"}
Aug 02 10:37:01.338936 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:37:01.338632    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4kwt9/perf-node-gather-daemonset-j25x6"
Aug 02 10:37:01.358599 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:37:01.358552    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4kwt9/perf-node-gather-daemonset-j25x6" podStartSLOduration=2.358538074 podStartE2EDuration="2.358538074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-02 10:36:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:37:01.356458181 +0000 UTC m=+3845.071263629" watchObservedRunningTime="2026-08-02 10:37:01.358538074 +0000 UTC m=+3845.073343559"
Aug 02 10:37:02.405982 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:37:02.405938    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7f8lc_6b6e3c57-2d9d-419a-a7e2-c419d29faecc/dns/0.log"
Aug 02 10:37:02.430008 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:37:02.429978    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7f8lc_6b6e3c57-2d9d-419a-a7e2-c419d29faecc/kube-rbac-proxy/0.log"
Aug 02 10:37:02.601402 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:37:02.601374    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4f6pd_032156ef-c7a4-4cd5-a991-4c91f2a67b4b/dns-node-resolver/0.log"
Aug 02 10:37:03.235790 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:37:03.235765    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8lm5w_6b0cc25e-0367-4ac6-b094-1caf4d2023ab/node-ca/0.log"
Aug 02 10:37:04.565493 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:37:04.565460    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-cbzmt_84d87887-e1dd-4a84-807d-6f551f7cbe8a/discovery/0.log"
Aug 02 10:37:04.704942 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:37:04.704914    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-hfw8f_d41871d5-cf2c-4676-8d98-96060bef7b10/payload-pre-processing/0.log"
Aug 02 10:37:04.743715 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:37:04.743691    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-7b8zc_9eda18ed-b699-4efd-b283-47c0ac424ecb/payload-processing/0.log"
Aug 02 10:37:05.675304 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:37:05.675262    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pvvv7_80c7ce46-ead1-48ad-9c47-296c0124131d/serve-healthcheck-canary/0.log"
Aug 02 10:37:06.220023 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:37:06.219994    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-89bsp_0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb/kube-rbac-proxy/0.log"
Aug 02 10:37:06.243764 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:37:06.243738    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-89bsp_0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb/exporter/0.log"
Aug 02 10:37:06.271815 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:37:06.271789    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-89bsp_0c4922a2-8f81-4ce6-8a51-4f7b727ba0eb/extractor/0.log"
Aug 02 10:37:07.352953 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:37:07.352926    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4kwt9/perf-node-gather-daemonset-j25x6"
Aug 02 10:37:09.635146 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:37:09.635106    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5694fb54d4-6z7v6_91e8b861-99fe-4311-9db5-ab8763266d31/manager/0.log"
Aug 02 10:37:09.663634 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:37:09.663595    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5694fb54d4-6z7v6_91e8b861-99fe-4311-9db5-ab8763266d31/copy-manifests/0.log"
Aug 02 10:37:10.875780 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:37:10.875745    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-cbzmt_84d87887-e1dd-4a84-807d-6f551f7cbe8a/discovery/0.log"
Aug 02 10:37:10.958346 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:37:10.958270    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-hfw8f_d41871d5-cf2c-4676-8d98-96060bef7b10/payload-pre-processing/0.log"
Aug 02 10:37:10.986358 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:37:10.986327    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-7b8zc_9eda18ed-b699-4efd-b283-47c0ac424ecb/payload-processing/0.log"
Aug 02 10:37:11.020225 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:37:11.020194    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-df7456b48-kdms8_678d742f-961f-4c9a-8a20-bdd3e807bbd6/manager/0.log"
Aug 02 10:37:12.744816 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:37:12.744785    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-cbzmt_84d87887-e1dd-4a84-807d-6f551f7cbe8a/discovery/0.log"
Aug 02 10:37:12.839259 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:37:12.838730    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-hfw8f_d41871d5-cf2c-4676-8d98-96060bef7b10/payload-pre-processing/0.log"
Aug 02 10:37:12.884429 ip-10-0-141-232 kubenswrapper[2564]: I0802 10:37:12.884400    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-7b8zc_9eda18ed-b699-4efd-b283-47c0ac424ecb/payload-processing/0.log"