Jul 15 20:45:35.102495 ip-10-0-137-234 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 15 20:45:35.102509 ip-10-0-137-234 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 15 20:45:35.102518 ip-10-0-137-234 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 15 20:45:35.102824 ip-10-0-137-234 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 15 20:45:45.216437 ip-10-0-137-234 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 15 20:45:45.216454 ip-10-0-137-234 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot d0890949fb1143dab25dd37658fa3029 --
Jul 15 20:47:46.240466 ip-10-0-137-234 systemd[1]: Starting Kubernetes Kubelet...
Jul 15 20:47:47.487582 ip-10-0-137-234 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 15 20:47:47.487582 ip-10-0-137-234 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 15 20:47:47.487582 ip-10-0-137-234 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 15 20:47:47.487582 ip-10-0-137-234 kubenswrapper[2566]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 15 20:47:47.487582 ip-10-0-137-234 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 15 20:47:47.488269 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.487623    2566 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 15 20:47:47.492434 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492416    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 15 20:47:47.492434 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492433    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 15 20:47:47.492434 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492436    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 15 20:47:47.492533 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492440    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 15 20:47:47.492533 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492444    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 15 20:47:47.492533 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492447    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 15 20:47:47.492533 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492450    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 15 20:47:47.492533 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492453    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 15 20:47:47.492533 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492456    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 15 20:47:47.492533 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492459    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 15 20:47:47.492533 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492461    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 15 20:47:47.492533 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492464    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 15 20:47:47.492533 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492467    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 15 20:47:47.492533 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492470    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 15 20:47:47.492533 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492473    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 15 20:47:47.492533 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492475    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 15 20:47:47.492533 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492478    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 15 20:47:47.492533 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492483    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 15 20:47:47.492533 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492488    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 15 20:47:47.492533 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492491    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 15 20:47:47.492533 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492494    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 15 20:47:47.492533 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492497    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 15 20:47:47.492533 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492500    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 15 20:47:47.493070 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492503    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 15 20:47:47.493070 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492506    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 15 20:47:47.493070 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492509    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 15 20:47:47.493070 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492511    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 15 20:47:47.493070 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492514    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 15 20:47:47.493070 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492521    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 15 20:47:47.493070 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492524    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 15 20:47:47.493070 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492527    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 15 20:47:47.493070 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492530    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 15 20:47:47.493070 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492532    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 15 20:47:47.493070 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492535    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 15 20:47:47.493070 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492539    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 15 20:47:47.493070 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492541    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 15 20:47:47.493070 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492545    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 15 20:47:47.493070 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492547    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 15 20:47:47.493070 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492550    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 15 20:47:47.493070 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492553    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 15 20:47:47.493070 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492556    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 15 20:47:47.493070 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492559    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 15 20:47:47.493575 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492564    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 15 20:47:47.493575 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492568    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 15 20:47:47.493575 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492571    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 15 20:47:47.493575 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492573    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 15 20:47:47.493575 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492576    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 15 20:47:47.493575 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492579    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 15 20:47:47.493575 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492582    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 15 20:47:47.493575 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492584    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 15 20:47:47.493575 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492587    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 15 20:47:47.493575 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492590    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 15 20:47:47.493575 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492592    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 15 20:47:47.493575 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492595    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 15 20:47:47.493575 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492598    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 15 20:47:47.493575 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492600    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 15 20:47:47.493575 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492604    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 15 20:47:47.493575 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492607    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 15 20:47:47.493575 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492610    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 15 20:47:47.493575 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492613    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 15 20:47:47.493575 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492617    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 15 20:47:47.493575 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492619    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 15 20:47:47.494118 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492622    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 15 20:47:47.494118 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492625    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 15 20:47:47.494118 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492628    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 15 20:47:47.494118 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492631    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 15 20:47:47.494118 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492633    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 15 20:47:47.494118 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492636    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 15 20:47:47.494118 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492639    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 15 20:47:47.494118 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492642    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 15 20:47:47.494118 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492644    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 15 20:47:47.494118 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492647    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 15 20:47:47.494118 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492650    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 15 20:47:47.494118 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492654    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 15 20:47:47.494118 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492658    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 15 20:47:47.494118 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492661    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 15 20:47:47.494118 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492664    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 15 20:47:47.494118 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492666    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 15 20:47:47.494118 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492671    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 15 20:47:47.494118 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492674    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 15 20:47:47.494118 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492677    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 15 20:47:47.494599 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492680    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 15 20:47:47.494599 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492682    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 15 20:47:47.494599 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492685    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 15 20:47:47.494599 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492688    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 15 20:47:47.494599 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492691    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 15 20:47:47.494599 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492693    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 15 20:47:47.494599 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492696    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 15 20:47:47.494599 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492699    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 15 20:47:47.494599 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492702    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 15 20:47:47.494599 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492705    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 15 20:47:47.494599 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492708    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 15 20:47:47.494599 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492711    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 15 20:47:47.494599 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492714    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 15 20:47:47.494599 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492717    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 15 20:47:47.494599 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492720    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 15 20:47:47.494599 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492723    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 15 20:47:47.494599 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492726    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 15 20:47:47.494599 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.492729    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 15 20:47:47.494599 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493527    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 15 20:47:47.494599 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493535    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 15 20:47:47.495108 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493539    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 15 20:47:47.495108 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493542    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 15 20:47:47.495108 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493545    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 15 20:47:47.495108 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493548    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 15 20:47:47.495108 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493553    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 15 20:47:47.495108 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493556    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 15 20:47:47.495108 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493559    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 15 20:47:47.495108 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493561    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 15 20:47:47.495108 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493564    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 15 20:47:47.495108 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493567    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 15 20:47:47.495108 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493570    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 15 20:47:47.495108 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493573    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 15 20:47:47.495108 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493579    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 15 20:47:47.495108 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493583    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 15 20:47:47.495108 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493586    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 15 20:47:47.495108 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493590    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 15 20:47:47.495108 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493593    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 15 20:47:47.495108 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493595    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 15 20:47:47.495108 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493598    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 15 20:47:47.495108 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493601    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 15 20:47:47.495664 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493604    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 15 20:47:47.495664 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493606    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 15 20:47:47.495664 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493610    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 15 20:47:47.495664 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493613    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 15 20:47:47.495664 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493616    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 15 20:47:47.495664 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493619    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 15 20:47:47.495664 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493622    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 15 20:47:47.495664 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493625    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 15 20:47:47.495664 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493628    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 15 20:47:47.495664 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493631    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 15 20:47:47.495664 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493633    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 15 20:47:47.495664 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493636    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 15 20:47:47.495664 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493639    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 15 20:47:47.495664 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493642    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 15 20:47:47.495664 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493644    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 15 20:47:47.495664 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493647    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 15 20:47:47.495664 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493650    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 15 20:47:47.495664 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493653    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 15 20:47:47.495664 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493655    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 15 20:47:47.495664 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493660    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 15 20:47:47.496210 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493663    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 15 20:47:47.496210 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493666    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 15 20:47:47.496210 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493669    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 15 20:47:47.496210 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493672    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 15 20:47:47.496210 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493675    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 15 20:47:47.496210 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493678    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 15 20:47:47.496210 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493681    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 15 20:47:47.496210 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493684    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 15 20:47:47.496210 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493686    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 15 20:47:47.496210 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493690    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 15 20:47:47.496210 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493692    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 15 20:47:47.496210 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493695    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 15 20:47:47.496210 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493697    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 15 20:47:47.496210 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493700    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 15 20:47:47.496210 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493703    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 15 20:47:47.496210 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493706    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 15 20:47:47.496210 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493709    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 15 20:47:47.496210 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493712    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 15 20:47:47.496210 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493714    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 15 20:47:47.496210 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493717    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 15 20:47:47.496712 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493720    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 15 20:47:47.496712 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493722    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 15 20:47:47.496712 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493725    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 15 20:47:47.496712 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493728    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 15 20:47:47.496712 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493730    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 15 20:47:47.496712 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493733    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 15 20:47:47.496712 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493736    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 15 20:47:47.496712 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493738    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 15 20:47:47.496712 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493741    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 15 20:47:47.496712 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493744    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 15 20:47:47.496712 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493748    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 15 20:47:47.496712 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493751    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 15 20:47:47.496712 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493754    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 15 20:47:47.496712 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493756    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 15 20:47:47.496712 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493759    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 15 20:47:47.496712 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493761    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 15 20:47:47.496712 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493764    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 15 20:47:47.496712 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493767    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 15 20:47:47.496712 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493770    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 15 20:47:47.496712 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493773    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 15 20:47:47.497265 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493775    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 15 20:47:47.497265 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493778    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 15 20:47:47.497265 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493781    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 15 20:47:47.497265 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493783    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 15 20:47:47.497265 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493786    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 15 20:47:47.497265 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493788    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 15 20:47:47.497265 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493791    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 15 20:47:47.497265 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493794    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 15 20:47:47.497265 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493797    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 15 20:47:47.497265 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493799    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 15 20:47:47.497265 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493802    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 15 20:47:47.497265 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493804    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 15 20:47:47.497265 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493807    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 15 20:47:47.497265 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493811    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 15 20:47:47.497265 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493815    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 15 20:47:47.497265 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493819    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 15 20:47:47.497265 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.493823    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 15 20:47:47.497265 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494245    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 15 20:47:47.497729 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494251    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 15 20:47:47.497729 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494255    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 15 20:47:47.497729 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494258    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 15 20:47:47.497729 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494261    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 15 20:47:47.497729 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494264    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 15 20:47:47.497729 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494267    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 15 20:47:47.497729 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494270    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 15 20:47:47.497729 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494273    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 15 20:47:47.497729 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494277    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 15 20:47:47.497729 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494280    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 15 20:47:47.497729 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494283    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 15 20:47:47.497729 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494286    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 15 20:47:47.497729 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494289    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 15 20:47:47.497729 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494292    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 15 20:47:47.497729 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494294    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 15 20:47:47.497729 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494297    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 15 20:47:47.497729 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494300    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 15 20:47:47.497729 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494302    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 15 20:47:47.497729 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494305    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 15 20:47:47.498244 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494308    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 15 20:47:47.498244 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494311    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 15 20:47:47.498244 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494314    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 15 20:47:47.498244 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494317    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 15 20:47:47.498244 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494320    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 15 20:47:47.498244 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494322    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 15 20:47:47.498244 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494325    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 15 20:47:47.498244 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494328    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 15 20:47:47.498244 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494330    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 15 20:47:47.498244 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494333    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 15 20:47:47.498244 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494336    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 15 20:47:47.498244 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494338    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 15 20:47:47.498244 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494341    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 15 20:47:47.498244 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494344    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 15 20:47:47.498244 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494347    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 15 20:47:47.498244 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494349    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 15 20:47:47.498244 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494352    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 15 20:47:47.498244 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494355    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 15 20:47:47.498244 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494360    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 15 20:47:47.498720 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494363    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 15 20:47:47.498720 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494366    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 15 20:47:47.498720 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494369    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 15 20:47:47.498720 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494372    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 15 20:47:47.498720 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494376    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 15 20:47:47.498720 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494379    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 15 20:47:47.498720 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494382    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 15 20:47:47.498720 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494386    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 15 20:47:47.498720 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494388    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 15 20:47:47.498720 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494391    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 15 20:47:47.498720 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494393    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 15 20:47:47.498720 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494396    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 15 20:47:47.498720 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494399    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 15 20:47:47.498720 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494401    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 15 20:47:47.498720 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494404    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 15 20:47:47.498720 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494407    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 15 20:47:47.498720 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494409    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 15 20:47:47.498720 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494412    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 15 20:47:47.498720 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494415    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 15 20:47:47.498720 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494417    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 15 20:47:47.499247 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494420    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 15 20:47:47.499247 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494422    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 15 20:47:47.499247 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494425    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 15 20:47:47.499247 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494427    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 15 20:47:47.499247 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494432    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 15 20:47:47.499247 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494435    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 15 20:47:47.499247 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494438    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 15 20:47:47.499247 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494441    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 15 20:47:47.499247 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494444    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 15 20:47:47.499247 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494447    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 15 20:47:47.499247 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494450    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 15 20:47:47.499247 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494453    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 15 20:47:47.499247 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494456    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 15 20:47:47.499247 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494458    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 15 20:47:47.499247 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494461    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 15 20:47:47.499247 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494464    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 15 20:47:47.499247 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494467    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 15 20:47:47.499247 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494470    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 15 20:47:47.499247 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494474    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 15 20:47:47.499729 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494477    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 15 20:47:47.499729 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494481    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 15 20:47:47.499729 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494484    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 15 20:47:47.499729 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494486    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 15 20:47:47.499729 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494489    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 15 20:47:47.499729 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494492    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 15 20:47:47.499729 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494495    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 15 20:47:47.499729 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494498    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 15 20:47:47.499729 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494500    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 15 20:47:47.499729 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494503    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 15 20:47:47.499729 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494505    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 15 20:47:47.499729 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494508    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 15 20:47:47.499729 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494511    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 15 20:47:47.499729 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494513    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 15 20:47:47.499729 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494516    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 15 20:47:47.499729 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494518    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 15 20:47:47.499729 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494521    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 15 20:47:47.499729 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494524    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 15 20:47:47.499729 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494527    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 15 20:47:47.499729 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494529    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 15 20:47:47.500275 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.494532    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 15 20:47:47.500275 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494609    2566 flags.go:64] FLAG: --address="0.0.0.0"
Jul 15 20:47:47.500275 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494621    2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 15 20:47:47.500275 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494631    2566 flags.go:64] FLAG: --anonymous-auth="true"
Jul 15 20:47:47.500275 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494636    2566 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 15 20:47:47.500275 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494640    2566 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 15 20:47:47.500275 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494644    2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 15 20:47:47.500275 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494648    2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 15 20:47:47.500275 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494653    2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 15 20:47:47.500275 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494656    2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 15 20:47:47.500275 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494662    2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 15 20:47:47.500275 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494665    2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 15 20:47:47.500275 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494669    2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 15 20:47:47.500275 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494672    2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 15 20:47:47.500275 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494676    2566 flags.go:64] FLAG: --cgroup-root=""
Jul 15 20:47:47.500275 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494679    2566 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 15 20:47:47.500275 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494683    2566 flags.go:64] FLAG: --client-ca-file=""
Jul 15 20:47:47.500275 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494685    2566 flags.go:64] FLAG: --cloud-provider="external"
Jul 15 20:47:47.500275 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494689    2566 flags.go:64] FLAG: --cluster-dns="[]"
Jul 15 20:47:47.500275 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494694    2566 flags.go:64] FLAG: --cluster-domain=""
Jul 15 20:47:47.500275 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494697    2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 15 20:47:47.500275 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494700    2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 15 20:47:47.500275 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494703    2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 15 20:47:47.500275 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494706    2566 flags.go:64] FLAG: --container-log-max-files="5"
Jul 15 20:47:47.500879 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494710    2566 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 15 20:47:47.500879 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494713    2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 15 20:47:47.500879 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494717    2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 15 20:47:47.500879 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494720    2566 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 15 20:47:47.500879 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494723    2566 flags.go:64] FLAG: --contention-profiling="false"
Jul 15 20:47:47.500879 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494726    2566 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 15 20:47:47.500879 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494729    2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 15 20:47:47.500879 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494733    2566 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 15 20:47:47.500879 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494736    2566 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 15 20:47:47.500879 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494742    2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 15 20:47:47.500879 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494746    2566 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 15 20:47:47.500879 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494750    2566 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 15 20:47:47.500879 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494753    2566 flags.go:64] FLAG: --enable-load-reader="false"
Jul 15 20:47:47.500879 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494756    2566 flags.go:64] FLAG: --enable-server="true"
Jul 15 20:47:47.500879 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494759    2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 15 20:47:47.500879 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494763    2566 flags.go:64] FLAG: --event-burst="100"
Jul 15 20:47:47.500879 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494767    2566 flags.go:64] FLAG: --event-qps="50"
Jul 15 20:47:47.500879 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494770    2566 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 15 20:47:47.500879 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494775    2566 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 15 20:47:47.500879 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494778    2566 flags.go:64] FLAG: --eviction-hard=""
Jul 15 20:47:47.500879 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494782    2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 15 20:47:47.500879 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494785    2566 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 15 20:47:47.500879 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494789    2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 15 20:47:47.500879 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494792    2566 flags.go:64] FLAG: --eviction-soft=""
Jul 15 20:47:47.500879 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494795    2566 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 15 20:47:47.501843 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494799    2566 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 15 20:47:47.501843 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494802    2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 15 20:47:47.501843 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494805    2566 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 15 20:47:47.501843 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494808    2566 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 15 20:47:47.501843 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494811    2566 flags.go:64] FLAG: --fail-swap-on="true"
Jul 15 20:47:47.501843 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494814    2566 flags.go:64] FLAG: --feature-gates=""
Jul 15 20:47:47.501843 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494818    2566 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 15 20:47:47.501843 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494821    2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 15 20:47:47.501843 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494825    2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 15 20:47:47.501843 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494828    2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 15 20:47:47.501843 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494831    2566 flags.go:64] FLAG: --healthz-port="10248"
Jul 15 20:47:47.501843 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494834    2566 flags.go:64] FLAG: --help="false"
Jul 15 20:47:47.501843 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494837    2566 flags.go:64] FLAG: --hostname-override="ip-10-0-137-234.ec2.internal"
Jul 15 20:47:47.501843 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494840    2566 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 15 20:47:47.501843 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494843    2566 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 15 20:47:47.501843 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494846    2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 15 20:47:47.501843 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494850    2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 15 20:47:47.501843 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494853    2566 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 15 20:47:47.501843 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494858    2566 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 15 20:47:47.501843 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494861    2566 flags.go:64] FLAG: --image-service-endpoint=""
Jul 15 20:47:47.501843 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494864    2566 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 15 20:47:47.501843 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494867    2566 flags.go:64] FLAG: --kube-api-burst="100"
Jul 15 20:47:47.501843 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494870    2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 15 20:47:47.503023 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494873    2566 flags.go:64] FLAG: --kube-api-qps="50"
Jul 15 20:47:47.503023 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494876    2566 flags.go:64] FLAG: --kube-reserved=""
Jul 15 20:47:47.503023 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494881    2566 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 15 20:47:47.503023 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494884    2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 15 20:47:47.503023 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494887    2566 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 15 20:47:47.503023 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494890    2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 15 20:47:47.503023 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494893    2566 flags.go:64] FLAG: --lock-file=""
Jul 15 20:47:47.503023 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494896    2566 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 15 20:47:47.503023 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494900    2566 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 15 20:47:47.503023 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494903    2566 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 15 20:47:47.503023 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494909    2566 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 15 20:47:47.503023 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494912    2566 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 15 20:47:47.503023 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494915    2566 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 15 20:47:47.503023 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494918    2566 flags.go:64] FLAG: --logging-format="text"
Jul 15 20:47:47.503023 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494921    2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 15 20:47:47.503023 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494924    2566 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 15 20:47:47.503023 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494927    2566 flags.go:64] FLAG: --manifest-url=""
Jul 15 20:47:47.503023 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494930    2566 flags.go:64] FLAG: --manifest-url-header=""
Jul 15 20:47:47.503023 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494935    2566 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 15 20:47:47.503023 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494938    2566 flags.go:64] FLAG: --max-open-files="1000000"
Jul 15 20:47:47.503023 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494942    2566 flags.go:64] FLAG: --max-pods="110"
Jul 15 20:47:47.503023 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494946    2566 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 15 20:47:47.503023 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494949    2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 15 20:47:47.503023 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494952    2566 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 15 20:47:47.503023 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494955    2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 15 20:47:47.504155 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494959    2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 15 20:47:47.504155 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494962    2566 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 15 20:47:47.504155 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494979    2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 15 20:47:47.504155 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494987    2566 flags.go:64] FLAG: --node-status-max-images="50"
Jul 15 20:47:47.504155 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494991    2566 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 15 20:47:47.504155 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494994    2566 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 15 20:47:47.504155 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.494997    2566 flags.go:64] FLAG: --pod-cidr=""
Jul 15 20:47:47.504155 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.495001    2566 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 15 20:47:47.504155 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.495006    2566 flags.go:64] FLAG: --pod-manifest-path=""
Jul 15 20:47:47.504155 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.495011    2566 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 15 20:47:47.504155 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.495014    2566 flags.go:64] FLAG: --pods-per-core="0"
Jul 15 20:47:47.504155 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.495017    2566 flags.go:64] FLAG: --port="10250"
Jul 15 20:47:47.504155 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.495021    2566 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 15 20:47:47.504155 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.495024    2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0275342c4fcafd29e"
Jul 15 20:47:47.504155 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.495027    2566 flags.go:64] FLAG: --qos-reserved=""
Jul 15 20:47:47.504155 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.495031    2566 flags.go:64] FLAG: --read-only-port="10255"
Jul 15 20:47:47.504155 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.495035    2566 flags.go:64] FLAG: --register-node="true"
Jul 15 20:47:47.504155 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.495038    2566 flags.go:64] FLAG: --register-with-taints=""
Jul 15 20:47:47.504155 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.495042    2566 flags.go:64] FLAG: --registry-burst="10"
Jul 15 20:47:47.504155 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.495045    2566 flags.go:64] FLAG: --registry-qps="5"
Jul 15 20:47:47.504155 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.495048    2566 flags.go:64] FLAG: --reserved-cpus=""
Jul 15 20:47:47.504155 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.495051    2566 flags.go:64] FLAG: --reserved-memory=""
Jul 15 20:47:47.504155 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.495055    2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 15 20:47:47.504155 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.495058    2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 15 20:47:47.504155 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.495061    2566 flags.go:64] FLAG: --rotate-certificates="false"
Jul 15 20:47:47.504801 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.495064    2566 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 15 20:47:47.504801 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.495067    2566 flags.go:64] FLAG: --runonce="false"
Jul 15 20:47:47.504801 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.495070    2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 15 20:47:47.504801 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.495074    2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 15 20:47:47.504801 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.495077    2566 flags.go:64] FLAG: --seccomp-default="false"
Jul 15 20:47:47.504801 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.495080    2566 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 15 20:47:47.504801 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.495083    2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 15 20:47:47.504801 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.495086    2566 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 15 20:47:47.504801 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.495089    2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 15 20:47:47.504801 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.495092    2566 flags.go:64] FLAG: --storage-driver-password="root"
Jul 15 20:47:47.504801 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.495097    2566 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 15 20:47:47.504801 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.495100    2566 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 15 20:47:47.504801 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.495103    2566 flags.go:64] FLAG: --storage-driver-user="root"
Jul 15 20:47:47.504801 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.495106    2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 15 20:47:47.504801 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.495109    2566 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 15 20:47:47.504801 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.495113    2566 flags.go:64] FLAG: --system-cgroups=""
Jul 15 20:47:47.504801 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.495117    2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 15 20:47:47.504801 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.495123    2566 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 15 20:47:47.504801 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.495126    2566 flags.go:64] FLAG: --tls-cert-file=""
Jul 15 20:47:47.504801 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.495129    2566 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 15 20:47:47.504801 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.495134    2566 flags.go:64] FLAG: --tls-min-version=""
Jul 15 20:47:47.504801 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.495137    2566 flags.go:64] FLAG: --tls-private-key-file=""
Jul 15 20:47:47.504801 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.495140    2566 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 15 20:47:47.504801 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.495143    2566 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 15 20:47:47.504801 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.495147    2566 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 15 20:47:47.505939 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.495150    2566 flags.go:64] FLAG: --v="2"
Jul 15 20:47:47.505939 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.495154    2566 flags.go:64] FLAG: --version="false"
Jul 15 20:47:47.505939 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.495158    2566 flags.go:64] FLAG: --vmodule=""
Jul 15 20:47:47.505939 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.495162    2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 15 20:47:47.505939 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.495165    2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 15 20:47:47.505939 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495311    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 15 20:47:47.505939 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495316    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 15 20:47:47.505939 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495319    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 15 20:47:47.505939 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495322    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 15 20:47:47.505939 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495325    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 15 20:47:47.505939 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495328    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 15 20:47:47.505939 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495331    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 15 20:47:47.505939 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495334    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 15 20:47:47.505939 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495336    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 15 20:47:47.505939 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495339    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 15 20:47:47.505939 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495342    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 15 20:47:47.505939 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495345    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 15 20:47:47.505939 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495349    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 15 20:47:47.505939 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495352    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 15 20:47:47.505939 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495355    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 15 20:47:47.505939 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495357    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 15 20:47:47.505939 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495360    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 15 20:47:47.506868 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495363    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 15 20:47:47.506868 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495367    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 15 20:47:47.506868 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495370    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 15 20:47:47.506868 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495373    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 15 20:47:47.506868 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495375    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 15 20:47:47.506868 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495378    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 15 20:47:47.506868 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495381    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 15 20:47:47.506868 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495384    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 15 20:47:47.506868 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495386    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 15 20:47:47.506868 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495389    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 15 20:47:47.506868 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495392    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 15 20:47:47.506868 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495395    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 15 20:47:47.506868 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495399    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 15 20:47:47.506868 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495403    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 15 20:47:47.506868 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495406    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 15 20:47:47.506868 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495409    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 15 20:47:47.506868 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495412    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 15 20:47:47.506868 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495415    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 15 20:47:47.506868 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495417    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 15 20:47:47.507481 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495420    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 15 20:47:47.507481 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495423    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 15 20:47:47.507481 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495426    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 15 20:47:47.507481 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495428    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 15 20:47:47.507481 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495431    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 15 20:47:47.507481 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495434    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 15 20:47:47.507481 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495436    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 15 20:47:47.507481 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495439    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 15 20:47:47.507481 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495443    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 15 20:47:47.507481 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495446    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 15 20:47:47.507481 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495449    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 15 20:47:47.507481 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495452    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 15 20:47:47.507481 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495455    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 15 20:47:47.507481 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495458    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 15 20:47:47.507481 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495462    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 15 20:47:47.507481 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495465    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 15 20:47:47.507481 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495468    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 15 20:47:47.507481 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495471    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 15 20:47:47.507481 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495473    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 15 20:47:47.507481 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495476    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 15 20:47:47.508428 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495478    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 15 20:47:47.508428 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495481    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 15 20:47:47.508428 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495484    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 15 20:47:47.508428 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495486    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 15 20:47:47.508428 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495489    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 15 20:47:47.508428 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495492    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 15 20:47:47.508428 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495495    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 15 20:47:47.508428 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495497    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 15 20:47:47.508428 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495500    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 15 20:47:47.508428 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495503    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 15 20:47:47.508428 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495506    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 15 20:47:47.508428 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495508    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 15 20:47:47.508428 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495511    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 15 20:47:47.508428 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495514    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 15 20:47:47.508428 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495516    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 15 20:47:47.508428 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495519    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 15 20:47:47.508428 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495522    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 15 20:47:47.508428 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495524    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 15 20:47:47.508428 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495527    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 15 20:47:47.508428 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495530    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 15 20:47:47.509229 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495534    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 15 20:47:47.509229 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495536    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 15 20:47:47.509229 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495539    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 15 20:47:47.509229 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495542    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 15 20:47:47.509229 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495545    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 15 20:47:47.509229 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495548    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 15 20:47:47.509229 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495552    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 15 20:47:47.509229 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495555    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 15 20:47:47.509229 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495557    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 15 20:47:47.509229 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495560    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 15 20:47:47.509229 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495563    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 15 20:47:47.509229 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495566    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 15 20:47:47.509229 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495568    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 15 20:47:47.509229 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495573    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 15 20:47:47.509229 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495576    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 15 20:47:47.509229 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495579    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 15 20:47:47.509229 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495582    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 15 20:47:47.509229 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495585    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 15 20:47:47.509229 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495588    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 15 20:47:47.509854 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495591    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 15 20:47:47.509854 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495595    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 15 20:47:47.509854 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495599    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 15 20:47:47.509854 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.495602    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 15 20:47:47.509854 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.495611    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 15 20:47:47.509854 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.504080    2566 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 15 20:47:47.509854 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.504102    2566 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 15 20:47:47.509854 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.505077    2566 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 15 20:47:47.509854 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.505092    2566 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 15 20:47:47.509854 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505195    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 15 20:47:47.509854 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505206    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 15 20:47:47.509854 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505211    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 15 20:47:47.509854 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505216    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 15 20:47:47.509854 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505221    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 15 20:47:47.510296 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505227    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 15 20:47:47.510296 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505232    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 15 20:47:47.510296 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505237    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 15 20:47:47.510296 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505241    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 15 20:47:47.510296 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505246    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 15 20:47:47.510296 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505251    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 15 20:47:47.510296 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505256    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 15 20:47:47.510296 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505261    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 15 20:47:47.510296 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505266    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 15 20:47:47.510296 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505270    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 15 20:47:47.510296 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505275    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 15 20:47:47.510296 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505282    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 15 20:47:47.510296 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505289    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 15 20:47:47.510296 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505294    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 15 20:47:47.510296 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505299    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 15 20:47:47.510296 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505304    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 15 20:47:47.510296 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505309    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 15 20:47:47.510296 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505314    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 15 20:47:47.510296 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505319    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 15 20:47:47.510296 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505323    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 15 20:47:47.510809 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505328    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 15 20:47:47.510809 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505333    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 15 20:47:47.510809 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505337    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 15 20:47:47.510809 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505341    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 15 20:47:47.510809 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505349    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 15 20:47:47.510809 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505354    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 15 20:47:47.510809 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505358    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 15 20:47:47.510809 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505363    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 15 20:47:47.510809 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505368    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 15 20:47:47.510809 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505372    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 15 20:47:47.510809 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505377    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 15 20:47:47.510809 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505382    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 15 20:47:47.510809 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505386    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 15 20:47:47.510809 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505391    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 15 20:47:47.510809 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505396    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 15 20:47:47.510809 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505400    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 15 20:47:47.510809 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505405    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 15 20:47:47.510809 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505409    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 15 20:47:47.510809 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505416    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 15 20:47:47.511384 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505422    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 15 20:47:47.511384 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505427    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 15 20:47:47.511384 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505432    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 15 20:47:47.511384 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505436    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 15 20:47:47.511384 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505442    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 15 20:47:47.511384 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505446    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 15 20:47:47.511384 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505451    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 15 20:47:47.511384 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505455    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 15 20:47:47.511384 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505460    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 15 20:47:47.511384 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505465    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 15 20:47:47.511384 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505470    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 15 20:47:47.511384 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505474    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 15 20:47:47.511384 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505479    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 15 20:47:47.511384 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505483    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 15 20:47:47.511384 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505488    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 15 20:47:47.511384 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505492    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 15 20:47:47.511384 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505497    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 15 20:47:47.511384 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505502    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 15 20:47:47.511384 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505507    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 15 20:47:47.511384 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505511    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 15 20:47:47.511952 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505516    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 15 20:47:47.511952 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505520    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 15 20:47:47.511952 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505525    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 15 20:47:47.511952 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505529    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 15 20:47:47.511952 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505534    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 15 20:47:47.511952 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505538    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 15 20:47:47.511952 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505543    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 15 20:47:47.511952 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505547    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 15 20:47:47.511952 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505551    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 15 20:47:47.511952 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505556    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 15 20:47:47.511952 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505560    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 15 20:47:47.511952 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505565    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 15 20:47:47.511952 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505569    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 15 20:47:47.511952 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505573    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 15 20:47:47.511952 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505578    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 15 20:47:47.511952 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505585    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 15 20:47:47.511952 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505591    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 15 20:47:47.511952 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505596    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 15 20:47:47.511952 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505601    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 15 20:47:47.512478 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505606    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 15 20:47:47.512478 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505611    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 15 20:47:47.512478 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505617    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 15 20:47:47.512478 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505622    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 15 20:47:47.512478 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505626    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 15 20:47:47.512478 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505631    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 15 20:47:47.512478 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505635    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 15 20:47:47.512478 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505640    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 15 20:47:47.512478 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505644    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 15 20:47:47.512478 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505648    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 15 20:47:47.512478 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505653    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 15 20:47:47.512478 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505658    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 15 20:47:47.512478 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505664    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 15 20:47:47.512478 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505668    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 15 20:47:47.512478 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505673    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 15 20:47:47.512478 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505678    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 15 20:47:47.512922 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.505687    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 15 20:47:47.512922 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505891    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 15 20:47:47.512922 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505900    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 15 20:47:47.512922 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505906    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 15 20:47:47.512922 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505911    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 15 20:47:47.512922 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505916    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 15 20:47:47.512922 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505921    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 15 20:47:47.512922 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505925    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 15 20:47:47.512922 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505930    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 15 20:47:47.512922 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505935    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 15 20:47:47.512922 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505939    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 15 20:47:47.512922 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505943    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 15 20:47:47.512922 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505949    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 15 20:47:47.512922 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505956    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 15 20:47:47.513316 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505983    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 15 20:47:47.513316 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505990    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 15 20:47:47.513316 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.505996    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 15 20:47:47.513316 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506002    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 15 20:47:47.513316 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506006    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 15 20:47:47.513316 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506011    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 15 20:47:47.513316 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506016    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 15 20:47:47.513316 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506021    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 15 20:47:47.513316 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506026    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 15 20:47:47.513316 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506030    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 15 20:47:47.513316 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506035    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 15 20:47:47.513316 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506039    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 15 20:47:47.513316 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506043    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 15 20:47:47.513316 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506048    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 15 20:47:47.513316 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506053    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 15 20:47:47.513316 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506059    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 15 20:47:47.513316 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506064    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 15 20:47:47.513316 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506068    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 15 20:47:47.513316 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506073    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 15 20:47:47.513316 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506077    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 15 20:47:47.514159 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506081    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 15 20:47:47.514159 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506086    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 15 20:47:47.514159 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506090    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 15 20:47:47.514159 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506094    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 15 20:47:47.514159 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506099    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 15 20:47:47.514159 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506103    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 15 20:47:47.514159 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506108    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 15 20:47:47.514159 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506114    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 15 20:47:47.514159 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506118    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 15 20:47:47.514159 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506123    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 15 20:47:47.514159 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506127    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 15 20:47:47.514159 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506132    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 15 20:47:47.514159 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506137    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 15 20:47:47.514159 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506141    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 15 20:47:47.514159 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506146    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 15 20:47:47.514159 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506150    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 15 20:47:47.514159 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506154    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 15 20:47:47.514159 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506159    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 15 20:47:47.514159 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506163    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 15 20:47:47.514159 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506167    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 15 20:47:47.514862 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506172    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 15 20:47:47.514862 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506176    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 15 20:47:47.514862 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506181    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 15 20:47:47.514862 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506186    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 15 20:47:47.514862 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506190    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 15 20:47:47.514862 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506195    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 15 20:47:47.514862 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506200    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 15 20:47:47.514862 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506204    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 15 20:47:47.514862 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506210    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 15 20:47:47.514862 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506217    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 15 20:47:47.514862 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506222    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 15 20:47:47.514862 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506227    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 15 20:47:47.514862 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506233    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 15 20:47:47.514862 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506238    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 15 20:47:47.514862 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506243    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 15 20:47:47.514862 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506248    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 15 20:47:47.514862 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506253    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 15 20:47:47.514862 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506257    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 15 20:47:47.514862 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506262    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 15 20:47:47.515447 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506267    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 15 20:47:47.515447 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506272    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 15 20:47:47.515447 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506277    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 15 20:47:47.515447 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506282    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 15 20:47:47.515447 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506286    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 15 20:47:47.515447 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506291    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 15 20:47:47.515447 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506295    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 15 20:47:47.515447 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506300    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 15 20:47:47.515447 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506304    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 15 20:47:47.515447 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506308    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 15 20:47:47.515447 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506313    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 15 20:47:47.515447 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506317    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 15 20:47:47.515447 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506322    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 15 20:47:47.515447 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506326    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 15 20:47:47.515447 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506331    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 15 20:47:47.515447 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506335    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 15 20:47:47.515447 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506340    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 15 20:47:47.515447 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506344    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 15 20:47:47.515447 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506348    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 15 20:47:47.515447 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506353    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 15 20:47:47.515447 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506357    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 15 20:47:47.516179 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506362    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 15 20:47:47.516179 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506367    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 15 20:47:47.516179 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506371    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 15 20:47:47.516179 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506375    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 15 20:47:47.516179 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506380    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 15 20:47:47.516179 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:47.506384    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 15 20:47:47.516179 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.506393    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 15 20:47:47.516179 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.506563    2566 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 15 20:47:47.516179 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.509460    2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 15 20:47:47.516179 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.510929    2566 server.go:1019] "Starting client certificate rotation"
Jul 15 20:47:47.516179 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.511047    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 15 20:47:47.516179 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.511081    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 15 20:47:47.516179 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.514187    2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 15 20:47:47.518202 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.518185    2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 15 20:47:47.529597 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.529572    2566 log.go:25] "Validated CRI v1 runtime API"
Jul 15 20:47:47.532173 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.532150    2566 log.go:25] "Validated CRI v1 image API"
Jul 15 20:47:47.533014 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.532996    2566 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 15 20:47:47.535917 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.535898    2566 fs.go:135] Filesystem UUIDs: map[3a781954-0437-466c-b981-d09aeac2acaf:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d29ae36c-008c-485e-8727-b576d4e978f5:/dev/nvme0n1p4]
Jul 15 20:47:47.535984 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.535916    2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 15 20:47:47.541347 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.541221    2566 manager.go:216] Machine: {Timestamp:2026-07-15 20:47:47.539744314 +0000 UTC m=+0.688782365 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099710 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29e06c2049fa1a2514a0f1ee41a6c2 SystemUUID:ec29e06c-2049-fa1a-2514-a0f1ee41a6c2 BootID:d0890949-fb11-43da-b25d-d37658fa3029 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ba:c9:59:79:c7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ba:c9:59:79:c7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7e:20:1e:7a:04:0d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 15 20:47:47.541347 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.541338    2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 15 20:47:47.541484 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.541457    2566 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 15 20:47:47.543237 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.543223    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 15 20:47:47.545324 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.545296    2566 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 15 20:47:47.545475 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.545328    2566 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-234.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 15 20:47:47.546065 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.546055    2566 topology_manager.go:138] "Creating topology manager with none policy"
Jul 15 20:47:47.546098 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.546068    2566 container_manager_linux.go:309] "Creating device plugin manager"
Jul 15 20:47:47.546098 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.546076    2566 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 15 20:47:47.546841 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.546830    2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 15 20:47:47.547770 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.547759    2566 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 15 20:47:47.550713 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.550701    2566 state_mem.go:36] "Initialized new in-memory state store"
Jul 15 20:47:47.550847 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.550837    2566 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 15 20:47:47.550880 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.550870    2566 kubelet.go:486] "Attempting to sync node with API server"
Jul 15 20:47:47.550908 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.550881    2566 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 15 20:47:47.550908 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.550892    2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 15 20:47:47.550908 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.550902    2566 kubelet.go:398] "Adding apiserver pod source"
Jul 15 20:47:47.551008 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.550911    2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 15 20:47:47.551358 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.551332    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 15 20:47:47.551414 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.551361    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 15 20:47:47.554268 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.554248    2566 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 15 20:47:47.556248 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.556233    2566 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 15 20:47:47.556300 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.556274    2566 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 15 20:47:47.557132 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.557118    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 15 20:47:47.557179 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.557144    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 15 20:47:47.557179 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.557158    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 15 20:47:47.557179 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.557172    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 15 20:47:47.557263 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.557204    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 15 20:47:47.557263 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.557217    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 15 20:47:47.557263 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.557235    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 15 20:47:47.557263 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.557249    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 15 20:47:47.557370 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.557264    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 15 20:47:47.557370 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.557307    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 15 20:47:47.557370 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.557335    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 15 20:47:47.557370 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.557348    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 15 20:47:47.558295 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.558283    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 15 20:47:47.558336 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.558296    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 15 20:47:47.562752 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.562724    2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-234.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 15 20:47:47.562846 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:47.562754    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-234.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 15 20:47:47.563944 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:47.563915    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 15 20:47:47.564046 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.564028    2566 server.go:1268] "Started kubelet"
Jul 15 20:47:47.564100 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.564076    2566 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 15 20:47:47.564310 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.564236    2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 15 20:47:47.564393 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.564367    2566 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 15 20:47:47.564968 ip-10-0-137-234 systemd[1]: Started Kubernetes Kubelet.
Jul 15 20:47:47.565646 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.565548    2566 server.go:310] "Adding debug handlers to kubelet server"
Jul 15 20:47:47.565963 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.565756    2566 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 15 20:47:47.569911 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.569893    2566 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 15 20:47:47.570025 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.569916    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 15 20:47:47.570277 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:47.570130    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-234.ec2.internal\" not found"
Jul 15 20:47:47.570569 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.570537    2566 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 15 20:47:47.570688 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.570678    2566 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 15 20:47:47.570752 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.570738    2566 factory.go:55] Registering systemd factory
Jul 15 20:47:47.570858 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.570754    2566 factory.go:223] Registration of the systemd container factory successfully
Jul 15 20:47:47.570858 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.570537    2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 15 20:47:47.570858 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.570812    2566 reconstruct.go:97] "Volume reconstruction finished"
Jul 15 20:47:47.570858 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.570822    2566 reconciler.go:29] "Reconciler: start to sync state"
Jul 15 20:47:47.571076 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:47.569827    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-234.ec2.internal.18c2919413b5b6af  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-234.ec2.internal,UID:ip-10-0-137-234.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-234.ec2.internal,},FirstTimestamp:2026-07-15 20:47:47.564000943 +0000 UTC m=+0.713038994,LastTimestamp:2026-07-15 20:47:47.564000943 +0000 UTC m=+0.713038994,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-234.ec2.internal,}"
Jul 15 20:47:47.571076 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.571003    2566 factory.go:153] Registering CRI-O factory
Jul 15 20:47:47.571076 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.571021    2566 factory.go:223] Registration of the crio container factory successfully
Jul 15 20:47:47.571225 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.571082    2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 15 20:47:47.571225 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.571108    2566 factory.go:103] Registering Raw factory
Jul 15 20:47:47.571225 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.571124    2566 manager.go:1196] Started watching for new ooms in manager
Jul 15 20:47:47.571594 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:47.571573    2566 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 15 20:47:47.571796 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.571782    2566 manager.go:349] Starting recovery of all containers
Jul 15 20:47:47.573332 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.573311    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zm877"
Jul 15 20:47:47.581490 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:47.581460    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 15 20:47:47.581580 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:47.581538    2566 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-234.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 15 20:47:47.582502 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.582470    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zm877"
Jul 15 20:47:47.583227 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.583208    2566 manager.go:354] Recovery completed
Jul 15 20:47:47.590367 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.590352    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 15 20:47:47.592425 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.592407    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-234.ec2.internal" event="NodeHasSufficientMemory"
Jul 15 20:47:47.592510 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.592443    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-234.ec2.internal" event="NodeHasNoDiskPressure"
Jul 15 20:47:47.592510 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.592455    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-234.ec2.internal" event="NodeHasSufficientPID"
Jul 15 20:47:47.593015 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.592998    2566 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 15 20:47:47.593015 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.593013    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 15 20:47:47.593131 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.593014    2566 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 15 20:47:47.593131 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.593086    2566 state_mem.go:36] "Initialized new in-memory state store"
Jul 15 20:47:47.593709 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.593694    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-234.ec2.internal" event="NodeHasSufficientMemory"
Jul 15 20:47:47.593786 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.593719    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-234.ec2.internal" event="NodeHasNoDiskPressure"
Jul 15 20:47:47.593786 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.593729    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-234.ec2.internal" event="NodeHasSufficientPID"
Jul 15 20:47:47.594354 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.594341    2566 policy_none.go:49] "None policy: Start"
Jul 15 20:47:47.594393 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.594358    2566 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 15 20:47:47.594393 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.594369    2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 15 20:47:47.595502 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.595491    2566 policy_none.go:47] "Start"
Jul 15 20:47:47.631746 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.631562    2566 manager.go:337] "Starting Device Plugin manager"
Jul 15 20:47:47.631838 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:47.631774    2566 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 15 20:47:47.631838 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.631790    2566 server.go:87] "Starting device plugin registration server"
Jul 15 20:47:47.632058 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.632043    2566 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 15 20:47:47.632097 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.632060    2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 15 20:47:47.632220 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.632193    2566 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 15 20:47:47.632816 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.632294    2566 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 15 20:47:47.632816 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.632302    2566 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 15 20:47:47.632974 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:47.632852    2566 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 15 20:47:47.632974 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:47.632912    2566 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-234.ec2.internal\" not found"
Jul 15 20:47:47.732627 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.732587    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 15 20:47:47.733793 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.733646    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-234.ec2.internal" event="NodeHasSufficientMemory"
Jul 15 20:47:47.733793 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.733686    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-234.ec2.internal" event="NodeHasNoDiskPressure"
Jul 15 20:47:47.733793 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.733698    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-234.ec2.internal" event="NodeHasSufficientPID"
Jul 15 20:47:47.733793 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.733727    2566 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-234.ec2.internal"
Jul 15 20:47:47.741516 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.741474    2566 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-234.ec2.internal"
Jul 15 20:47:47.741516 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:47.741500    2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-234.ec2.internal\": node \"ip-10-0-137-234.ec2.internal\" not found"
Jul 15 20:47:47.748830 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.748803    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 15 20:47:47.750154 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.750128    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 15 20:47:47.750242 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.750174    2566 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 15 20:47:47.750242 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.750199    2566 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 15 20:47:47.750343 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:47.750240    2566 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 15 20:47:47.752349 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.752322    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 15 20:47:47.757874 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:47.757853    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-234.ec2.internal\" not found"
Jul 15 20:47:47.853030 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.852973    2566 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-137-234.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-234.ec2.internal"]
Jul 15 20:47:47.853186 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.853066    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 15 20:47:47.854251 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.854233    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-234.ec2.internal" event="NodeHasSufficientMemory"
Jul 15 20:47:47.854322 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.854265    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-234.ec2.internal" event="NodeHasNoDiskPressure"
Jul 15 20:47:47.854322 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.854277    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-234.ec2.internal" event="NodeHasSufficientPID"
Jul 15 20:47:47.855639 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.855624    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 15 20:47:47.855770 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.855752    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-234.ec2.internal"
Jul 15 20:47:47.855821 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.855795    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 15 20:47:47.856465 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.856447    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-234.ec2.internal" event="NodeHasSufficientMemory"
Jul 15 20:47:47.856567 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.856466    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-234.ec2.internal" event="NodeHasSufficientMemory"
Jul 15 20:47:47.856567 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.856476    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-234.ec2.internal" event="NodeHasNoDiskPressure"
Jul 15 20:47:47.856567 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.856486    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-234.ec2.internal" event="NodeHasSufficientPID"
Jul 15 20:47:47.856567 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.856489    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-234.ec2.internal" event="NodeHasNoDiskPressure"
Jul 15 20:47:47.856567 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.856500    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-234.ec2.internal" event="NodeHasSufficientPID"
Jul 15 20:47:47.857846 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.857832    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-234.ec2.internal"
Jul 15 20:47:47.857910 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.857861    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 15 20:47:47.857910 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:47.857890    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-234.ec2.internal\" not found"
Jul 15 20:47:47.858603 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.858585    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-234.ec2.internal" event="NodeHasSufficientMemory"
Jul 15 20:47:47.858694 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.858622    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-234.ec2.internal" event="NodeHasNoDiskPressure"
Jul 15 20:47:47.858694 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.858640    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-234.ec2.internal" event="NodeHasSufficientPID"
Jul 15 20:47:47.872481 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.872458    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b740363c6967fb438e6610aa1d6ea6c6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-234.ec2.internal\" (UID: \"b740363c6967fb438e6610aa1d6ea6c6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-234.ec2.internal"
Jul 15 20:47:47.872568 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.872487    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4a1659746931c9e82b46a4829b7de6fa-config\") pod \"kube-apiserver-proxy-ip-10-0-137-234.ec2.internal\" (UID: \"4a1659746931c9e82b46a4829b7de6fa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-234.ec2.internal"
Jul 15 20:47:47.872568 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.872505    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b740363c6967fb438e6610aa1d6ea6c6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-234.ec2.internal\" (UID: \"b740363c6967fb438e6610aa1d6ea6c6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-234.ec2.internal"
Jul 15 20:47:47.884716 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:47.884694    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-234.ec2.internal\" not found" node="ip-10-0-137-234.ec2.internal"
Jul 15 20:47:47.889373 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:47.889357    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-234.ec2.internal\" not found" node="ip-10-0-137-234.ec2.internal"
Jul 15 20:47:47.958672 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:47.958637    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-234.ec2.internal\" not found"
Jul 15 20:47:47.973052 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.973020    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4a1659746931c9e82b46a4829b7de6fa-config\") pod \"kube-apiserver-proxy-ip-10-0-137-234.ec2.internal\" (UID: \"4a1659746931c9e82b46a4829b7de6fa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-234.ec2.internal"
Jul 15 20:47:47.973052 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.973050    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b740363c6967fb438e6610aa1d6ea6c6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-234.ec2.internal\" (UID: \"b740363c6967fb438e6610aa1d6ea6c6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-234.ec2.internal"
Jul 15 20:47:47.973198 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.973068    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b740363c6967fb438e6610aa1d6ea6c6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-234.ec2.internal\" (UID: \"b740363c6967fb438e6610aa1d6ea6c6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-234.ec2.internal"
Jul 15 20:47:47.973198 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.973119    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b740363c6967fb438e6610aa1d6ea6c6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-234.ec2.internal\" (UID: \"b740363c6967fb438e6610aa1d6ea6c6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-234.ec2.internal"
Jul 15 20:47:47.973198 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.973129    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4a1659746931c9e82b46a4829b7de6fa-config\") pod \"kube-apiserver-proxy-ip-10-0-137-234.ec2.internal\" (UID: \"4a1659746931c9e82b46a4829b7de6fa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-234.ec2.internal"
Jul 15 20:47:47.973198 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:47.973148    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b740363c6967fb438e6610aa1d6ea6c6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-234.ec2.internal\" (UID: \"b740363c6967fb438e6610aa1d6ea6c6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-234.ec2.internal"
Jul 15 20:47:48.058965 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:48.058870    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-234.ec2.internal\" not found"
Jul 15 20:47:48.162185 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:48.162131    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-234.ec2.internal\" not found"
Jul 15 20:47:48.187448 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:48.187415    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-234.ec2.internal"
Jul 15 20:47:48.192888 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:48.192868    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-234.ec2.internal"
Jul 15 20:47:48.262825 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:48.262788    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-234.ec2.internal\" not found"
Jul 15 20:47:48.367162 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:48.367075    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-234.ec2.internal\" not found"
Jul 15 20:47:48.450100 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:48.450072    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 15 20:47:48.467556 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:48.467526    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-234.ec2.internal\" not found"
Jul 15 20:47:48.510445 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:48.510401    2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 15 20:47:48.510899 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:48.510573    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 15 20:47:48.510899 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:48.510637    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 15 20:47:48.568123 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:48.568096    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-234.ec2.internal\" not found"
Jul 15 20:47:48.589376 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:48.589338    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-14 20:42:47 +0000 UTC" deadline="2028-02-25 02:47:25.528744096 +0000 UTC"
Jul 15 20:47:48.589376 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:48.589365    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14141h59m36.939382817s"
Jul 15 20:47:48.668643 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:48.668622    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-234.ec2.internal\" not found"
Jul 15 20:47:48.766422 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:48.766388    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a1659746931c9e82b46a4829b7de6fa.slice/crio-288b12ae98ad360d5c76857cac760c5cc0e3863420c4f1536f5ef2bd97916d29 WatchSource:0}: Error finding container 288b12ae98ad360d5c76857cac760c5cc0e3863420c4f1536f5ef2bd97916d29: Status 404 returned error can't find the container with id 288b12ae98ad360d5c76857cac760c5cc0e3863420c4f1536f5ef2bd97916d29
Jul 15 20:47:48.766637 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:48.766613    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb740363c6967fb438e6610aa1d6ea6c6.slice/crio-d94935ff9f777da69b329ed5aea91e25be055afc97c47639f7441fe31d2c64c8 WatchSource:0}: Error finding container d94935ff9f777da69b329ed5aea91e25be055afc97c47639f7441fe31d2c64c8: Status 404 returned error can't find the container with id d94935ff9f777da69b329ed5aea91e25be055afc97c47639f7441fe31d2c64c8
Jul 15 20:47:48.768747 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:48.768727    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-234.ec2.internal\" not found"
Jul 15 20:47:48.774227 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:48.774210    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 15 20:47:48.869613 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:48.869566    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-234.ec2.internal\" not found"
Jul 15 20:47:48.977312 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:48.977224    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-234.ec2.internal\" not found"
Jul 15 20:47:49.077445 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:49.077414    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-234.ec2.internal\" not found"
Jul 15 20:47:49.129415 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.129386    2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 15 20:47:49.152019 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.151975    2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 15 20:47:49.170398 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.170358    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-234.ec2.internal"
Jul 15 20:47:49.184244 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.184220    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 15 20:47:49.185405 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.185392    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-234.ec2.internal"
Jul 15 20:47:49.197312 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.197298    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 15 20:47:49.421455 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.421356    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 15 20:47:49.553124 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.553083    2566 apiserver.go:52] "Watching apiserver"
Jul 15 20:47:49.561924 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.561891    2566 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 15 20:47:49.562445 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.562415    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-wvh9c","openshift-multus/network-metrics-daemon-xwbql","openshift-ovn-kubernetes/ovnkube-node-652lk","kube-system/global-pull-secret-syncer-zzw8h","kube-system/konnectivity-agent-6j89c","kube-system/kube-apiserver-proxy-ip-10-0-137-234.ec2.internal","openshift-dns/node-resolver-qvvx2","openshift-image-registry/node-ca-wcchl","openshift-network-diagnostics/network-check-target-ngpfr","openshift-network-operator/iptables-alerter-n2vtn","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8twzl","openshift-cluster-node-tuning-operator/tuned-7wpzz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-234.ec2.internal","openshift-multus/multus-additional-cni-plugins-wjxpz"]
Jul 15 20:47:49.564323 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.564299    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wjxpz"
Jul 15 20:47:49.565693 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.565673    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwbql"
Jul 15 20:47:49.565811 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:49.565754    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwbql" podUID="42a42d42-83b0-441b-b7f2-3770f7523317"
Jul 15 20:47:49.567006 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.566979    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qvvx2"
Jul 15 20:47:49.567930 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.567892    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vxg4d\""
Jul 15 20:47:49.568683 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.568322    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 15 20:47:49.568683 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.568579    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzw8h"
Jul 15 20:47:49.568858 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:49.568723    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzw8h" podUID="b9d9267e-0c23-42b2-bb48-55ed14050fff"
Jul 15 20:47:49.569482 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.569048    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 15 20:47:49.569482 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.569384    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 15 20:47:49.570142 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.569810    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 15 20:47:49.570142 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.570013    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 15 20:47:49.570692 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.570631    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6j89c"
Jul 15 20:47:49.570692 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.570653    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rnmn2\""
Jul 15 20:47:49.570929 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.570850    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 15 20:47:49.572728 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.571150    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 15 20:47:49.572728 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.571525    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 15 20:47:49.572728 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.572414    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.573493 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.573436    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wcchl"
Jul 15 20:47:49.576080 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.574776    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-wn45h\""
Jul 15 20:47:49.576080 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.574882    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 15 20:47:49.576080 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.575399    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngpfr"
Jul 15 20:47:49.576080 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:49.575484    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngpfr" podUID="55baaf18-8e1a-4d1b-9411-ed55205cc56c"
Jul 15 20:47:49.576080 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.575897    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 15 20:47:49.576362 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.576141    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 15 20:47:49.576496 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.576468    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-f8wqk\""
Jul 15 20:47:49.577180 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.577156    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-n2vtn"
Jul 15 20:47:49.579183 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.578746    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8twzl"
Jul 15 20:47:49.579822 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.579618    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 15 20:47:49.579822 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.579664    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 15 20:47:49.580013 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.579925    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 15 20:47:49.580013 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.579987    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ft2pv\""
Jul 15 20:47:49.580289 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.580269    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7wpzz"
Jul 15 20:47:49.581928 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.581910    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.582802 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.582346    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 15 20:47:49.582802 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.582603    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-j29r7\""
Jul 15 20:47:49.584036 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.583294    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2dbf2724-61f8-4572-a2c1-906032db929d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wjxpz\" (UID: \"2dbf2724-61f8-4572-a2c1-906032db929d\") " pod="openshift-multus/multus-additional-cni-plugins-wjxpz"
Jul 15 20:47:49.584036 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.583365    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmggz\" (UniqueName: \"kubernetes.io/projected/42a42d42-83b0-441b-b7f2-3770f7523317-kube-api-access-lmggz\") pod \"network-metrics-daemon-xwbql\" (UID: \"42a42d42-83b0-441b-b7f2-3770f7523317\") " pod="openshift-multus/network-metrics-daemon-xwbql"
Jul 15 20:47:49.584036 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.583401    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d786587a-4e93-443c-bfaa-334ced76d272-multus-socket-dir-parent\") pod \"multus-wvh9c\" (UID: \"d786587a-4e93-443c-bfaa-334ced76d272\") " pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.584036 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.583439    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2dbf2724-61f8-4572-a2c1-906032db929d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wjxpz\" (UID: \"2dbf2724-61f8-4572-a2c1-906032db929d\") " pod="openshift-multus/multus-additional-cni-plugins-wjxpz"
Jul 15 20:47:49.584036 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.583474    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9d9267e-0c23-42b2-bb48-55ed14050fff-original-pull-secret\") pod \"global-pull-secret-syncer-zzw8h\" (UID: \"b9d9267e-0c23-42b2-bb48-55ed14050fff\") " pod="kube-system/global-pull-secret-syncer-zzw8h"
Jul 15 20:47:49.584036 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.583507    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ad23d165-aeec-409f-99bd-12c27029221b-konnectivity-ca\") pod \"konnectivity-agent-6j89c\" (UID: \"ad23d165-aeec-409f-99bd-12c27029221b\") " pod="kube-system/konnectivity-agent-6j89c"
Jul 15 20:47:49.584036 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.583541    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d786587a-4e93-443c-bfaa-334ced76d272-system-cni-dir\") pod \"multus-wvh9c\" (UID: \"d786587a-4e93-443c-bfaa-334ced76d272\") " pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.584036 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.583567    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d786587a-4e93-443c-bfaa-334ced76d272-host-run-netns\") pod \"multus-wvh9c\" (UID: \"d786587a-4e93-443c-bfaa-334ced76d272\") " pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.585848 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.585394    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7l685\""
Jul 15 20:47:49.585848 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.585481    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 15 20:47:49.586030 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.585923    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 15 20:47:49.586833 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.586177    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 15 20:47:49.586833 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.586269    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 15 20:47:49.586833 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.586547    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-djkxp\""
Jul 15 20:47:49.586833 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.586610    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 15 20:47:49.586833 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.586609    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 15 20:47:49.586833 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.586546    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 15 20:47:49.587188 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.586867    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-v5f2t\""
Jul 15 20:47:49.587188 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.586717    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 15 20:47:49.587288 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.587213    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 15 20:47:49.589282 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.587379    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d786587a-4e93-443c-bfaa-334ced76d272-host-var-lib-cni-bin\") pod \"multus-wvh9c\" (UID: \"d786587a-4e93-443c-bfaa-334ced76d272\") " pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.589282 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.587438    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xsnx\" (UniqueName: \"kubernetes.io/projected/2dbf2724-61f8-4572-a2c1-906032db929d-kube-api-access-6xsnx\") pod \"multus-additional-cni-plugins-wjxpz\" (UID: \"2dbf2724-61f8-4572-a2c1-906032db929d\") " pod="openshift-multus/multus-additional-cni-plugins-wjxpz"
Jul 15 20:47:49.589282 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.587477    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42a42d42-83b0-441b-b7f2-3770f7523317-metrics-certs\") pod \"network-metrics-daemon-xwbql\" (UID: \"42a42d42-83b0-441b-b7f2-3770f7523317\") " pod="openshift-multus/network-metrics-daemon-xwbql"
Jul 15 20:47:49.589282 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.587506    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d786587a-4e93-443c-bfaa-334ced76d272-os-release\") pod \"multus-wvh9c\" (UID: \"d786587a-4e93-443c-bfaa-334ced76d272\") " pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.589282 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.587538    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d786587a-4e93-443c-bfaa-334ced76d272-host-run-k8s-cni-cncf-io\") pod \"multus-wvh9c\" (UID: \"d786587a-4e93-443c-bfaa-334ced76d272\") " pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.589282 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.587602    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d786587a-4e93-443c-bfaa-334ced76d272-hostroot\") pod \"multus-wvh9c\" (UID: \"d786587a-4e93-443c-bfaa-334ced76d272\") " pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.589282 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.587637    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d786587a-4e93-443c-bfaa-334ced76d272-multus-conf-dir\") pod \"multus-wvh9c\" (UID: \"d786587a-4e93-443c-bfaa-334ced76d272\") " pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.589282 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.587680    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d786587a-4e93-443c-bfaa-334ced76d272-multus-daemon-config\") pod \"multus-wvh9c\" (UID: \"d786587a-4e93-443c-bfaa-334ced76d272\") " pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.589282 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.587715    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d786587a-4e93-443c-bfaa-334ced76d272-host-run-multus-certs\") pod \"multus-wvh9c\" (UID: \"d786587a-4e93-443c-bfaa-334ced76d272\") " pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.589282 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.587748    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2dbf2724-61f8-4572-a2c1-906032db929d-system-cni-dir\") pod \"multus-additional-cni-plugins-wjxpz\" (UID: \"2dbf2724-61f8-4572-a2c1-906032db929d\") " pod="openshift-multus/multus-additional-cni-plugins-wjxpz"
Jul 15 20:47:49.589282 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.587762    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 15 20:47:49.589282 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.587778    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2dbf2724-61f8-4572-a2c1-906032db929d-os-release\") pod \"multus-additional-cni-plugins-wjxpz\" (UID: \"2dbf2724-61f8-4572-a2c1-906032db929d\") " pod="openshift-multus/multus-additional-cni-plugins-wjxpz"
Jul 15 20:47:49.589282 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.587803    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/147d4538-e5f1-47af-b61c-f07de8e0b0df-tmp-dir\") pod \"node-resolver-qvvx2\" (UID: \"147d4538-e5f1-47af-b61c-f07de8e0b0df\") " pod="openshift-dns/node-resolver-qvvx2"
Jul 15 20:47:49.589282 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.587832    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d786587a-4e93-443c-bfaa-334ced76d272-cnibin\") pod \"multus-wvh9c\" (UID: \"d786587a-4e93-443c-bfaa-334ced76d272\") " pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.589282 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.587862    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d786587a-4e93-443c-bfaa-334ced76d272-cni-binary-copy\") pod \"multus-wvh9c\" (UID: \"d786587a-4e93-443c-bfaa-334ced76d272\") " pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.589282 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.587893    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b9d9267e-0c23-42b2-bb48-55ed14050fff-dbus\") pod \"global-pull-secret-syncer-zzw8h\" (UID: \"b9d9267e-0c23-42b2-bb48-55ed14050fff\") " pod="kube-system/global-pull-secret-syncer-zzw8h"
Jul 15 20:47:49.589282 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.587926    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 15 20:47:49.589282 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.587949    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d786587a-4e93-443c-bfaa-334ced76d272-host-var-lib-cni-multus\") pod \"multus-wvh9c\" (UID: \"d786587a-4e93-443c-bfaa-334ced76d272\") " pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.590306 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.587982    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2dbf2724-61f8-4572-a2c1-906032db929d-cni-binary-copy\") pod \"multus-additional-cni-plugins-wjxpz\" (UID: \"2dbf2724-61f8-4572-a2c1-906032db929d\") " pod="openshift-multus/multus-additional-cni-plugins-wjxpz"
Jul 15 20:47:49.590306 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.588014    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm9zp\" (UniqueName: \"kubernetes.io/projected/147d4538-e5f1-47af-b61c-f07de8e0b0df-kube-api-access-lm9zp\") pod \"node-resolver-qvvx2\" (UID: \"147d4538-e5f1-47af-b61c-f07de8e0b0df\") " pod="openshift-dns/node-resolver-qvvx2"
Jul 15 20:47:49.590306 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.588043    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b9d9267e-0c23-42b2-bb48-55ed14050fff-kubelet-config\") pod \"global-pull-secret-syncer-zzw8h\" (UID: \"b9d9267e-0c23-42b2-bb48-55ed14050fff\") " pod="kube-system/global-pull-secret-syncer-zzw8h"
Jul 15 20:47:49.590306 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.588073    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2dbf2724-61f8-4572-a2c1-906032db929d-cnibin\") pod \"multus-additional-cni-plugins-wjxpz\" (UID: \"2dbf2724-61f8-4572-a2c1-906032db929d\") " pod="openshift-multus/multus-additional-cni-plugins-wjxpz"
Jul 15 20:47:49.590306 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.588096    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/147d4538-e5f1-47af-b61c-f07de8e0b0df-hosts-file\") pod \"node-resolver-qvvx2\" (UID: \"147d4538-e5f1-47af-b61c-f07de8e0b0df\") " pod="openshift-dns/node-resolver-qvvx2"
Jul 15 20:47:49.590306 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.588126    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d786587a-4e93-443c-bfaa-334ced76d272-multus-cni-dir\") pod \"multus-wvh9c\" (UID: \"d786587a-4e93-443c-bfaa-334ced76d272\") " pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.590306 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.588159    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d786587a-4e93-443c-bfaa-334ced76d272-host-var-lib-kubelet\") pod \"multus-wvh9c\" (UID: \"d786587a-4e93-443c-bfaa-334ced76d272\") " pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.590306 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.588217    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94wgd\" (UniqueName: \"kubernetes.io/projected/d786587a-4e93-443c-bfaa-334ced76d272-kube-api-access-94wgd\") pod \"multus-wvh9c\" (UID: \"d786587a-4e93-443c-bfaa-334ced76d272\") " pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.590306 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.588250    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2dbf2724-61f8-4572-a2c1-906032db929d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wjxpz\" (UID: \"2dbf2724-61f8-4572-a2c1-906032db929d\") " pod="openshift-multus/multus-additional-cni-plugins-wjxpz"
Jul 15 20:47:49.590306 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.588513    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ad23d165-aeec-409f-99bd-12c27029221b-agent-certs\") pod \"konnectivity-agent-6j89c\" (UID: \"ad23d165-aeec-409f-99bd-12c27029221b\") " pod="kube-system/konnectivity-agent-6j89c"
Jul 15 20:47:49.590306 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.588551    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d786587a-4e93-443c-bfaa-334ced76d272-etc-kubernetes\") pod \"multus-wvh9c\" (UID: \"d786587a-4e93-443c-bfaa-334ced76d272\") " pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.590306 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.589003    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 15 20:47:49.590306 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.589032    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 15 20:47:49.592729 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.592683    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 15 20:47:49.629950 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.629903    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wkkfw"
Jul 15 20:47:49.639277 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.639240    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wkkfw"
Jul 15 20:47:49.671976 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.671881    2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 15 20:47:49.688839 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.688806    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d786587a-4e93-443c-bfaa-334ced76d272-cnibin\") pod \"multus-wvh9c\" (UID: \"d786587a-4e93-443c-bfaa-334ced76d272\") " pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.689017 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.688852    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d786587a-4e93-443c-bfaa-334ced76d272-cni-binary-copy\") pod \"multus-wvh9c\" (UID: \"d786587a-4e93-443c-bfaa-334ced76d272\") " pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.689017 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.688886    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e9ca36a8-de2e-472f-8b76-e6c4433f3ce4-sys-fs\") pod \"aws-ebs-csi-driver-node-8twzl\" (UID: \"e9ca36a8-de2e-472f-8b76-e6c4433f3ce4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8twzl"
Jul 15 20:47:49.689017 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.688947    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d786587a-4e93-443c-bfaa-334ced76d272-host-var-lib-cni-multus\") pod \"multus-wvh9c\" (UID: \"d786587a-4e93-443c-bfaa-334ced76d272\") " pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.689017 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.688977    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f9a159bb-39c4-4cca-92ff-1c74d739e311-host-run-netns\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.689017 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.688981    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d786587a-4e93-443c-bfaa-334ced76d272-cnibin\") pod \"multus-wvh9c\" (UID: \"d786587a-4e93-443c-bfaa-334ced76d272\") " pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.689017 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.689003    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f9a159bb-39c4-4cca-92ff-1c74d739e311-log-socket\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.689335 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.689042    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f9a159bb-39c4-4cca-92ff-1c74d739e311-ovnkube-script-lib\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.689335 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.689069    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d786587a-4e93-443c-bfaa-334ced76d272-host-var-lib-cni-multus\") pod \"multus-wvh9c\" (UID: \"d786587a-4e93-443c-bfaa-334ced76d272\") " pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.689335 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.689073    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pbmw\" (UniqueName: \"kubernetes.io/projected/1bfac382-72fc-400b-ab8b-9e4f53ab1d50-kube-api-access-4pbmw\") pod \"node-ca-wcchl\" (UID: \"1bfac382-72fc-400b-ab8b-9e4f53ab1d50\") " pod="openshift-image-registry/node-ca-wcchl"
Jul 15 20:47:49.689335 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.689115    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9a159bb-39c4-4cca-92ff-1c74d739e311-etc-openvswitch\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.689335 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.689158    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c252p\" (UniqueName: \"kubernetes.io/projected/f9a159bb-39c4-4cca-92ff-1c74d739e311-kube-api-access-c252p\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.689335 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.689197    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/147d4538-e5f1-47af-b61c-f07de8e0b0df-hosts-file\") pod \"node-resolver-qvvx2\" (UID: \"147d4538-e5f1-47af-b61c-f07de8e0b0df\") " pod="openshift-dns/node-resolver-qvvx2"
Jul 15 20:47:49.689335 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.689233    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d786587a-4e93-443c-bfaa-334ced76d272-host-var-lib-kubelet\") pod \"multus-wvh9c\" (UID: \"d786587a-4e93-443c-bfaa-334ced76d272\") " pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.689335 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.689257    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f9a159bb-39c4-4cca-92ff-1c74d739e311-run-ovn\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.689335 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.689285    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2dbf2724-61f8-4572-a2c1-906032db929d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wjxpz\" (UID: \"2dbf2724-61f8-4572-a2c1-906032db929d\") " pod="openshift-multus/multus-additional-cni-plugins-wjxpz"
Jul 15 20:47:49.689335 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.689311    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d786587a-4e93-443c-bfaa-334ced76d272-etc-kubernetes\") pod \"multus-wvh9c\" (UID: \"d786587a-4e93-443c-bfaa-334ced76d272\") " pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.689885 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.689385    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1bfac382-72fc-400b-ab8b-9e4f53ab1d50-serviceca\") pod \"node-ca-wcchl\" (UID: \"1bfac382-72fc-400b-ab8b-9e4f53ab1d50\") " pod="openshift-image-registry/node-ca-wcchl"
Jul 15 20:47:49.689885 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.689411    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1bbd7d75-7a89-403d-983b-6534ab13e786-etc-sysconfig\") pod \"tuned-7wpzz\" (UID: \"1bbd7d75-7a89-403d-983b-6534ab13e786\") " pod="openshift-cluster-node-tuning-operator/tuned-7wpzz"
Jul 15 20:47:49.689885 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.689426    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d786587a-4e93-443c-bfaa-334ced76d272-etc-kubernetes\") pod \"multus-wvh9c\" (UID: \"d786587a-4e93-443c-bfaa-334ced76d272\") " pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.689885 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.689436    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1bbd7d75-7a89-403d-983b-6534ab13e786-etc-sysctl-d\") pod \"tuned-7wpzz\" (UID: \"1bbd7d75-7a89-403d-983b-6534ab13e786\") " pod="openshift-cluster-node-tuning-operator/tuned-7wpzz"
Jul 15 20:47:49.689885 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.689476    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d786587a-4e93-443c-bfaa-334ced76d272-host-var-lib-kubelet\") pod \"multus-wvh9c\" (UID: \"d786587a-4e93-443c-bfaa-334ced76d272\") " pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.689885 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.689481    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1bbd7d75-7a89-403d-983b-6534ab13e786-etc-sysctl-conf\") pod \"tuned-7wpzz\" (UID: \"1bbd7d75-7a89-403d-983b-6534ab13e786\") " pod="openshift-cluster-node-tuning-operator/tuned-7wpzz"
Jul 15 20:47:49.689885 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.689733    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f9a159bb-39c4-4cca-92ff-1c74d739e311-host-slash\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.689885 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.689792    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f9a159bb-39c4-4cca-92ff-1c74d739e311-node-log\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.689885 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.689816    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1bbd7d75-7a89-403d-983b-6534ab13e786-etc-modprobe-d\") pod \"tuned-7wpzz\" (UID: \"1bbd7d75-7a89-403d-983b-6534ab13e786\") " pod="openshift-cluster-node-tuning-operator/tuned-7wpzz"
Jul 15 20:47:49.689885 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.689840    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1bbd7d75-7a89-403d-983b-6534ab13e786-sys\") pod \"tuned-7wpzz\" (UID: \"1bbd7d75-7a89-403d-983b-6534ab13e786\") " pod="openshift-cluster-node-tuning-operator/tuned-7wpzz"
Jul 15 20:47:49.689885 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.689891    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2dbf2724-61f8-4572-a2c1-906032db929d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wjxpz\" (UID: \"2dbf2724-61f8-4572-a2c1-906032db929d\") " pod="openshift-multus/multus-additional-cni-plugins-wjxpz"
Jul 15 20:47:49.690925 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.689932    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d786587a-4e93-443c-bfaa-334ced76d272-host-run-netns\") pod \"multus-wvh9c\" (UID: \"d786587a-4e93-443c-bfaa-334ced76d272\") " pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.690925 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.689962    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/147d4538-e5f1-47af-b61c-f07de8e0b0df-hosts-file\") pod \"node-resolver-qvvx2\" (UID: \"147d4538-e5f1-47af-b61c-f07de8e0b0df\") " pod="openshift-dns/node-resolver-qvvx2"
Jul 15 20:47:49.690925 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.690030    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d786587a-4e93-443c-bfaa-334ced76d272-host-var-lib-cni-bin\") pod \"multus-wvh9c\" (UID: \"d786587a-4e93-443c-bfaa-334ced76d272\") " pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.690925 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.690057    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d786587a-4e93-443c-bfaa-334ced76d272-host-run-multus-certs\") pod \"multus-wvh9c\" (UID: \"d786587a-4e93-443c-bfaa-334ced76d272\") " pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.690925 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.690088    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e9ca36a8-de2e-472f-8b76-e6c4433f3ce4-socket-dir\") pod \"aws-ebs-csi-driver-node-8twzl\" (UID: \"e9ca36a8-de2e-472f-8b76-e6c4433f3ce4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8twzl"
Jul 15 20:47:49.690925 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.690116    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j84nq\" (UniqueName: \"kubernetes.io/projected/e9ca36a8-de2e-472f-8b76-e6c4433f3ce4-kube-api-access-j84nq\") pod \"aws-ebs-csi-driver-node-8twzl\" (UID: \"e9ca36a8-de2e-472f-8b76-e6c4433f3ce4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8twzl"
Jul 15 20:47:49.690925 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.690114    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2dbf2724-61f8-4572-a2c1-906032db929d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wjxpz\" (UID: \"2dbf2724-61f8-4572-a2c1-906032db929d\") " pod="openshift-multus/multus-additional-cni-plugins-wjxpz"
Jul 15 20:47:49.690925 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.690151    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2dbf2724-61f8-4572-a2c1-906032db929d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wjxpz\" (UID: \"2dbf2724-61f8-4572-a2c1-906032db929d\") " pod="openshift-multus/multus-additional-cni-plugins-wjxpz"
Jul 15 20:47:49.690925 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.690201    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d786587a-4e93-443c-bfaa-334ced76d272-host-run-netns\") pod \"multus-wvh9c\" (UID: \"d786587a-4e93-443c-bfaa-334ced76d272\") " pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.690925 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.690208    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9a159bb-39c4-4cca-92ff-1c74d739e311-run-openvswitch\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.690925 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.690206    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d786587a-4e93-443c-bfaa-334ced76d272-host-var-lib-cni-bin\") pod \"multus-wvh9c\" (UID: \"d786587a-4e93-443c-bfaa-334ced76d272\") " pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.690925 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.690214    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d786587a-4e93-443c-bfaa-334ced76d272-cni-binary-copy\") pod \"multus-wvh9c\" (UID: \"d786587a-4e93-443c-bfaa-334ced76d272\") " pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.690925 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.690238    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d786587a-4e93-443c-bfaa-334ced76d272-host-run-multus-certs\") pod \"multus-wvh9c\" (UID: \"d786587a-4e93-443c-bfaa-334ced76d272\") " pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.690925 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.690269    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42a42d42-83b0-441b-b7f2-3770f7523317-metrics-certs\") pod \"network-metrics-daemon-xwbql\" (UID: \"42a42d42-83b0-441b-b7f2-3770f7523317\") " pod="openshift-multus/network-metrics-daemon-xwbql"
Jul 15 20:47:49.690925 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.690301    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d786587a-4e93-443c-bfaa-334ced76d272-os-release\") pod \"multus-wvh9c\" (UID: \"d786587a-4e93-443c-bfaa-334ced76d272\") " pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.690925 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.690350    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d786587a-4e93-443c-bfaa-334ced76d272-host-run-k8s-cni-cncf-io\") pod \"multus-wvh9c\" (UID: \"d786587a-4e93-443c-bfaa-334ced76d272\") " pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.690925 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.690380    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d786587a-4e93-443c-bfaa-334ced76d272-os-release\") pod \"multus-wvh9c\" (UID: \"d786587a-4e93-443c-bfaa-334ced76d272\") " pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.690925 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:49.690428    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 20:47:49.691841 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.690471    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d786587a-4e93-443c-bfaa-334ced76d272-multus-daemon-config\") pod \"multus-wvh9c\" (UID: \"d786587a-4e93-443c-bfaa-334ced76d272\") " pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.691841 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.690504    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d786587a-4e93-443c-bfaa-334ced76d272-host-run-k8s-cni-cncf-io\") pod \"multus-wvh9c\" (UID: \"d786587a-4e93-443c-bfaa-334ced76d272\") " pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.691841 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.690514    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3166ef5d-e008-44a0-90e4-4f9c39f2332d-iptables-alerter-script\") pod \"iptables-alerter-n2vtn\" (UID: \"3166ef5d-e008-44a0-90e4-4f9c39f2332d\") " pod="openshift-network-operator/iptables-alerter-n2vtn"
Jul 15 20:47:49.691841 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.690560    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9a159bb-39c4-4cca-92ff-1c74d739e311-host-kubelet\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.691841 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.690589    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9a159bb-39c4-4cca-92ff-1c74d739e311-var-lib-openvswitch\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.691841 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.690618    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f9a159bb-39c4-4cca-92ff-1c74d739e311-env-overrides\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.691841 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.690674    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2dbf2724-61f8-4572-a2c1-906032db929d-system-cni-dir\") pod \"multus-additional-cni-plugins-wjxpz\" (UID: \"2dbf2724-61f8-4572-a2c1-906032db929d\") " pod="openshift-multus/multus-additional-cni-plugins-wjxpz"
Jul 15 20:47:49.691841 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.690702    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2dbf2724-61f8-4572-a2c1-906032db929d-os-release\") pod \"multus-additional-cni-plugins-wjxpz\" (UID: \"2dbf2724-61f8-4572-a2c1-906032db929d\") " pod="openshift-multus/multus-additional-cni-plugins-wjxpz"
Jul 15 20:47:49.691841 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.690729    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/147d4538-e5f1-47af-b61c-f07de8e0b0df-tmp-dir\") pod \"node-resolver-qvvx2\" (UID: \"147d4538-e5f1-47af-b61c-f07de8e0b0df\") " pod="openshift-dns/node-resolver-qvvx2"
Jul 15 20:47:49.691841 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.690757    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpb6j\" (UniqueName: \"kubernetes.io/projected/55baaf18-8e1a-4d1b-9411-ed55205cc56c-kube-api-access-gpb6j\") pod \"network-check-target-ngpfr\" (UID: \"55baaf18-8e1a-4d1b-9411-ed55205cc56c\") " pod="openshift-network-diagnostics/network-check-target-ngpfr"
Jul 15 20:47:49.691841 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.690784    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bbd7d75-7a89-403d-983b-6534ab13e786-etc-kubernetes\") pod \"tuned-7wpzz\" (UID: \"1bbd7d75-7a89-403d-983b-6534ab13e786\") " pod="openshift-cluster-node-tuning-operator/tuned-7wpzz"
Jul 15 20:47:49.691841 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:49.690978    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42a42d42-83b0-441b-b7f2-3770f7523317-metrics-certs podName:42a42d42-83b0-441b-b7f2-3770f7523317 nodeName:}" failed. No retries permitted until 2026-07-15 20:47:50.190935482 +0000 UTC m=+3.339973538 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/42a42d42-83b0-441b-b7f2-3770f7523317-metrics-certs") pod "network-metrics-daemon-xwbql" (UID: "42a42d42-83b0-441b-b7f2-3770f7523317") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 20:47:49.691841 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.690983    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2dbf2724-61f8-4572-a2c1-906032db929d-system-cni-dir\") pod \"multus-additional-cni-plugins-wjxpz\" (UID: \"2dbf2724-61f8-4572-a2c1-906032db929d\") " pod="openshift-multus/multus-additional-cni-plugins-wjxpz"
Jul 15 20:47:49.691841 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.691036    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d786587a-4e93-443c-bfaa-334ced76d272-multus-daemon-config\") pod \"multus-wvh9c\" (UID: \"d786587a-4e93-443c-bfaa-334ced76d272\") " pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.691841 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.691043    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2dbf2724-61f8-4572-a2c1-906032db929d-os-release\") pod \"multus-additional-cni-plugins-wjxpz\" (UID: \"2dbf2724-61f8-4572-a2c1-906032db929d\") " pod="openshift-multus/multus-additional-cni-plugins-wjxpz"
Jul 15 20:47:49.691841 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.691111    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1bbd7d75-7a89-403d-983b-6534ab13e786-run\") pod \"tuned-7wpzz\" (UID: \"1bbd7d75-7a89-403d-983b-6534ab13e786\") " pod="openshift-cluster-node-tuning-operator/tuned-7wpzz"
Jul 15 20:47:49.692685 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.691151    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1bbd7d75-7a89-403d-983b-6534ab13e786-tmp\") pod \"tuned-7wpzz\" (UID: \"1bbd7d75-7a89-403d-983b-6534ab13e786\") " pod="openshift-cluster-node-tuning-operator/tuned-7wpzz"
Jul 15 20:47:49.692685 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.691192    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f9a159bb-39c4-4cca-92ff-1c74d739e311-run-systemd\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.692685 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.691237    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b9d9267e-0c23-42b2-bb48-55ed14050fff-dbus\") pod \"global-pull-secret-syncer-zzw8h\" (UID: \"b9d9267e-0c23-42b2-bb48-55ed14050fff\") " pod="kube-system/global-pull-secret-syncer-zzw8h"
Jul 15 20:47:49.692685 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.691288    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1bbd7d75-7a89-403d-983b-6534ab13e786-lib-modules\") pod \"tuned-7wpzz\" (UID: \"1bbd7d75-7a89-403d-983b-6534ab13e786\") " pod="openshift-cluster-node-tuning-operator/tuned-7wpzz"
Jul 15 20:47:49.692685 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.691299    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/147d4538-e5f1-47af-b61c-f07de8e0b0df-tmp-dir\") pod \"node-resolver-qvvx2\" (UID: \"147d4538-e5f1-47af-b61c-f07de8e0b0df\") " pod="openshift-dns/node-resolver-qvvx2"
Jul 15 20:47:49.692685 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.691315    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bbd7d75-7a89-403d-983b-6534ab13e786-var-lib-kubelet\") pod \"tuned-7wpzz\" (UID: \"1bbd7d75-7a89-403d-983b-6534ab13e786\") " pod="openshift-cluster-node-tuning-operator/tuned-7wpzz"
Jul 15 20:47:49.692685 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.691363    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9a159bb-39c4-4cca-92ff-1c74d739e311-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.692685 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.691389    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f9a159bb-39c4-4cca-92ff-1c74d739e311-ovnkube-config\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.692685 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.691416    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f9a159bb-39c4-4cca-92ff-1c74d739e311-ovn-node-metrics-cert\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.692685 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.691464    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2dbf2724-61f8-4572-a2c1-906032db929d-cni-binary-copy\") pod \"multus-additional-cni-plugins-wjxpz\" (UID: \"2dbf2724-61f8-4572-a2c1-906032db929d\") " pod="openshift-multus/multus-additional-cni-plugins-wjxpz"
Jul 15 20:47:49.692685 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.691471    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b9d9267e-0c23-42b2-bb48-55ed14050fff-dbus\") pod \"global-pull-secret-syncer-zzw8h\" (UID: \"b9d9267e-0c23-42b2-bb48-55ed14050fff\") " pod="kube-system/global-pull-secret-syncer-zzw8h"
Jul 15 20:47:49.692685 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.691509    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm9zp\" (UniqueName: \"kubernetes.io/projected/147d4538-e5f1-47af-b61c-f07de8e0b0df-kube-api-access-lm9zp\") pod \"node-resolver-qvvx2\" (UID: \"147d4538-e5f1-47af-b61c-f07de8e0b0df\") " pod="openshift-dns/node-resolver-qvvx2"
Jul 15 20:47:49.692685 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.691542    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b9d9267e-0c23-42b2-bb48-55ed14050fff-kubelet-config\") pod \"global-pull-secret-syncer-zzw8h\" (UID: \"b9d9267e-0c23-42b2-bb48-55ed14050fff\") " pod="kube-system/global-pull-secret-syncer-zzw8h"
Jul 15 20:47:49.692685 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.691573    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fpps\" (UniqueName: \"kubernetes.io/projected/3166ef5d-e008-44a0-90e4-4f9c39f2332d-kube-api-access-4fpps\") pod \"iptables-alerter-n2vtn\" (UID: \"3166ef5d-e008-44a0-90e4-4f9c39f2332d\") " pod="openshift-network-operator/iptables-alerter-n2vtn"
Jul 15 20:47:49.692685 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.691625    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e9ca36a8-de2e-472f-8b76-e6c4433f3ce4-registration-dir\") pod \"aws-ebs-csi-driver-node-8twzl\" (UID: \"e9ca36a8-de2e-472f-8b76-e6c4433f3ce4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8twzl"
Jul 15 20:47:49.692685 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.691655    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2dbf2724-61f8-4572-a2c1-906032db929d-cnibin\") pod \"multus-additional-cni-plugins-wjxpz\" (UID: \"2dbf2724-61f8-4572-a2c1-906032db929d\") " pod="openshift-multus/multus-additional-cni-plugins-wjxpz"
Jul 15 20:47:49.692685 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.691683    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d786587a-4e93-443c-bfaa-334ced76d272-multus-cni-dir\") pod \"multus-wvh9c\" (UID: \"d786587a-4e93-443c-bfaa-334ced76d272\") " pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.693466 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.691719    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b9d9267e-0c23-42b2-bb48-55ed14050fff-kubelet-config\") pod \"global-pull-secret-syncer-zzw8h\" (UID: \"b9d9267e-0c23-42b2-bb48-55ed14050fff\") " pod="kube-system/global-pull-secret-syncer-zzw8h"
Jul 15 20:47:49.693466 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.691765    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2dbf2724-61f8-4572-a2c1-906032db929d-cnibin\") pod \"multus-additional-cni-plugins-wjxpz\" (UID: \"2dbf2724-61f8-4572-a2c1-906032db929d\") " pod="openshift-multus/multus-additional-cni-plugins-wjxpz"
Jul 15 20:47:49.693466 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.691791    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d786587a-4e93-443c-bfaa-334ced76d272-multus-cni-dir\") pod \"multus-wvh9c\" (UID: \"d786587a-4e93-443c-bfaa-334ced76d272\") " pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.693466 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.691790    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94wgd\" (UniqueName: \"kubernetes.io/projected/d786587a-4e93-443c-bfaa-334ced76d272-kube-api-access-94wgd\") pod \"multus-wvh9c\" (UID: \"d786587a-4e93-443c-bfaa-334ced76d272\") " pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.693466 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.691940    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3166ef5d-e008-44a0-90e4-4f9c39f2332d-host-slash\") pod \"iptables-alerter-n2vtn\" (UID: \"3166ef5d-e008-44a0-90e4-4f9c39f2332d\") " pod="openshift-network-operator/iptables-alerter-n2vtn"
Jul 15 20:47:49.693466 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.691973    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e9ca36a8-de2e-472f-8b76-e6c4433f3ce4-device-dir\") pod \"aws-ebs-csi-driver-node-8twzl\" (UID: \"e9ca36a8-de2e-472f-8b76-e6c4433f3ce4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8twzl"
Jul 15 20:47:49.693466 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.692013    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2dbf2724-61f8-4572-a2c1-906032db929d-cni-binary-copy\") pod \"multus-additional-cni-plugins-wjxpz\" (UID: \"2dbf2724-61f8-4572-a2c1-906032db929d\") " pod="openshift-multus/multus-additional-cni-plugins-wjxpz"
Jul 15 20:47:49.693466 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.692022    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1bbd7d75-7a89-403d-983b-6534ab13e786-etc-tuned\") pod \"tuned-7wpzz\" (UID: \"1bbd7d75-7a89-403d-983b-6534ab13e786\") " pod="openshift-cluster-node-tuning-operator/tuned-7wpzz"
Jul 15 20:47:49.693466 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.692064    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f9a159bb-39c4-4cca-92ff-1c74d739e311-systemd-units\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.693466 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.692091    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9a159bb-39c4-4cca-92ff-1c74d739e311-host-run-ovn-kubernetes\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.693466 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.692128    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ad23d165-aeec-409f-99bd-12c27029221b-agent-certs\") pod \"konnectivity-agent-6j89c\" (UID: \"ad23d165-aeec-409f-99bd-12c27029221b\") " pod="kube-system/konnectivity-agent-6j89c"
Jul 15 20:47:49.693466 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.692240    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e9ca36a8-de2e-472f-8b76-e6c4433f3ce4-etc-selinux\") pod \"aws-ebs-csi-driver-node-8twzl\" (UID: \"e9ca36a8-de2e-472f-8b76-e6c4433f3ce4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8twzl"
Jul 15 20:47:49.693466 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.692268    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f9a159bb-39c4-4cca-92ff-1c74d739e311-host-cni-bin\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.693466 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.692307    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2dbf2724-61f8-4572-a2c1-906032db929d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wjxpz\" (UID: \"2dbf2724-61f8-4572-a2c1-906032db929d\") " pod="openshift-multus/multus-additional-cni-plugins-wjxpz"
Jul 15 20:47:49.693466 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.692341    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmggz\" (UniqueName: \"kubernetes.io/projected/42a42d42-83b0-441b-b7f2-3770f7523317-kube-api-access-lmggz\") pod \"network-metrics-daemon-xwbql\" (UID: \"42a42d42-83b0-441b-b7f2-3770f7523317\") " pod="openshift-multus/network-metrics-daemon-xwbql"
Jul 15 20:47:49.693466 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.692369    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e9ca36a8-de2e-472f-8b76-e6c4433f3ce4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8twzl\" (UID: \"e9ca36a8-de2e-472f-8b76-e6c4433f3ce4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8twzl"
Jul 15 20:47:49.694208 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.692394    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1bbd7d75-7a89-403d-983b-6534ab13e786-etc-systemd\") pod \"tuned-7wpzz\" (UID: \"1bbd7d75-7a89-403d-983b-6534ab13e786\") " pod="openshift-cluster-node-tuning-operator/tuned-7wpzz"
Jul 15 20:47:49.694208 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.692418    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1bbd7d75-7a89-403d-983b-6534ab13e786-host\") pod \"tuned-7wpzz\" (UID: \"1bbd7d75-7a89-403d-983b-6534ab13e786\") " pod="openshift-cluster-node-tuning-operator/tuned-7wpzz"
Jul 15 20:47:49.694208 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.692443    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrng5\" (UniqueName: \"kubernetes.io/projected/1bbd7d75-7a89-403d-983b-6534ab13e786-kube-api-access-nrng5\") pod \"tuned-7wpzz\" (UID: \"1bbd7d75-7a89-403d-983b-6534ab13e786\") " pod="openshift-cluster-node-tuning-operator/tuned-7wpzz"
Jul 15 20:47:49.694208 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.692466    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f9a159bb-39c4-4cca-92ff-1c74d739e311-host-cni-netd\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.694208 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.692496    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d786587a-4e93-443c-bfaa-334ced76d272-multus-socket-dir-parent\") pod \"multus-wvh9c\" (UID: \"d786587a-4e93-443c-bfaa-334ced76d272\") " pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.694208 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.692494    2566 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 15 20:47:49.694208 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.692524    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9d9267e-0c23-42b2-bb48-55ed14050fff-original-pull-secret\") pod \"global-pull-secret-syncer-zzw8h\" (UID: \"b9d9267e-0c23-42b2-bb48-55ed14050fff\") " pod="kube-system/global-pull-secret-syncer-zzw8h"
Jul 15 20:47:49.694208 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.692551    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ad23d165-aeec-409f-99bd-12c27029221b-konnectivity-ca\") pod \"konnectivity-agent-6j89c\" (UID: \"ad23d165-aeec-409f-99bd-12c27029221b\") " pod="kube-system/konnectivity-agent-6j89c"
Jul 15 20:47:49.694208 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.692575    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d786587a-4e93-443c-bfaa-334ced76d272-system-cni-dir\") pod \"multus-wvh9c\" (UID: \"d786587a-4e93-443c-bfaa-334ced76d272\") " pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.694208 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.692614    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1bfac382-72fc-400b-ab8b-9e4f53ab1d50-host\") pod \"node-ca-wcchl\" (UID: \"1bfac382-72fc-400b-ab8b-9e4f53ab1d50\") " pod="openshift-image-registry/node-ca-wcchl"
Jul 15 20:47:49.694208 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.692651    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d786587a-4e93-443c-bfaa-334ced76d272-multus-socket-dir-parent\") pod \"multus-wvh9c\" (UID: \"d786587a-4e93-443c-bfaa-334ced76d272\") " pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.694208 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.692655    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xsnx\" (UniqueName: \"kubernetes.io/projected/2dbf2724-61f8-4572-a2c1-906032db929d-kube-api-access-6xsnx\") pod \"multus-additional-cni-plugins-wjxpz\" (UID: \"2dbf2724-61f8-4572-a2c1-906032db929d\") " pod="openshift-multus/multus-additional-cni-plugins-wjxpz"
Jul 15 20:47:49.694208 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:49.692736    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 15 20:47:49.694208 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:49.692802    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9d9267e-0c23-42b2-bb48-55ed14050fff-original-pull-secret podName:b9d9267e-0c23-42b2-bb48-55ed14050fff nodeName:}" failed. No retries permitted until 2026-07-15 20:47:50.192777515 +0000 UTC m=+3.341815557 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b9d9267e-0c23-42b2-bb48-55ed14050fff-original-pull-secret") pod "global-pull-secret-syncer-zzw8h" (UID: "b9d9267e-0c23-42b2-bb48-55ed14050fff") : object "kube-system"/"original-pull-secret" not registered
Jul 15 20:47:49.694208 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.692949    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d786587a-4e93-443c-bfaa-334ced76d272-system-cni-dir\") pod \"multus-wvh9c\" (UID: \"d786587a-4e93-443c-bfaa-334ced76d272\") " pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.694208 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.692988    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d786587a-4e93-443c-bfaa-334ced76d272-hostroot\") pod \"multus-wvh9c\" (UID: \"d786587a-4e93-443c-bfaa-334ced76d272\") " pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.694208 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.693019    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d786587a-4e93-443c-bfaa-334ced76d272-multus-conf-dir\") pod \"multus-wvh9c\" (UID: \"d786587a-4e93-443c-bfaa-334ced76d272\") " pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.695064 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.693139    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d786587a-4e93-443c-bfaa-334ced76d272-multus-conf-dir\") pod \"multus-wvh9c\" (UID: \"d786587a-4e93-443c-bfaa-334ced76d272\") " pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.695064 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.693180    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d786587a-4e93-443c-bfaa-334ced76d272-hostroot\") pod \"multus-wvh9c\" (UID: \"d786587a-4e93-443c-bfaa-334ced76d272\") " pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.695064 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.693539    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2dbf2724-61f8-4572-a2c1-906032db929d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wjxpz\" (UID: \"2dbf2724-61f8-4572-a2c1-906032db929d\") " pod="openshift-multus/multus-additional-cni-plugins-wjxpz"
Jul 15 20:47:49.695064 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.693955    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ad23d165-aeec-409f-99bd-12c27029221b-konnectivity-ca\") pod \"konnectivity-agent-6j89c\" (UID: \"ad23d165-aeec-409f-99bd-12c27029221b\") " pod="kube-system/konnectivity-agent-6j89c"
Jul 15 20:47:49.696967 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.696948    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ad23d165-aeec-409f-99bd-12c27029221b-agent-certs\") pod \"konnectivity-agent-6j89c\" (UID: \"ad23d165-aeec-409f-99bd-12c27029221b\") " pod="kube-system/konnectivity-agent-6j89c"
Jul 15 20:47:49.706602 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.706567    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm9zp\" (UniqueName: \"kubernetes.io/projected/147d4538-e5f1-47af-b61c-f07de8e0b0df-kube-api-access-lm9zp\") pod \"node-resolver-qvvx2\" (UID: \"147d4538-e5f1-47af-b61c-f07de8e0b0df\") " pod="openshift-dns/node-resolver-qvvx2"
Jul 15 20:47:49.707476 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.707401    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xsnx\" (UniqueName: \"kubernetes.io/projected/2dbf2724-61f8-4572-a2c1-906032db929d-kube-api-access-6xsnx\") pod \"multus-additional-cni-plugins-wjxpz\" (UID: \"2dbf2724-61f8-4572-a2c1-906032db929d\") " pod="openshift-multus/multus-additional-cni-plugins-wjxpz"
Jul 15 20:47:49.707930 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.707884    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmggz\" (UniqueName: \"kubernetes.io/projected/42a42d42-83b0-441b-b7f2-3770f7523317-kube-api-access-lmggz\") pod \"network-metrics-daemon-xwbql\" (UID: \"42a42d42-83b0-441b-b7f2-3770f7523317\") " pod="openshift-multus/network-metrics-daemon-xwbql"
Jul 15 20:47:49.707930 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.707901    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94wgd\" (UniqueName: \"kubernetes.io/projected/d786587a-4e93-443c-bfaa-334ced76d272-kube-api-access-94wgd\") pod \"multus-wvh9c\" (UID: \"d786587a-4e93-443c-bfaa-334ced76d272\") " pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.755995 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.755931    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-234.ec2.internal" event={"ID":"b740363c6967fb438e6610aa1d6ea6c6","Type":"ContainerStarted","Data":"d94935ff9f777da69b329ed5aea91e25be055afc97c47639f7441fe31d2c64c8"}
Jul 15 20:47:49.757085 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.757052    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-234.ec2.internal" event={"ID":"4a1659746931c9e82b46a4829b7de6fa","Type":"ContainerStarted","Data":"288b12ae98ad360d5c76857cac760c5cc0e3863420c4f1536f5ef2bd97916d29"}
Jul 15 20:47:49.794035 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.794003    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1bfac382-72fc-400b-ab8b-9e4f53ab1d50-serviceca\") pod \"node-ca-wcchl\" (UID: \"1bfac382-72fc-400b-ab8b-9e4f53ab1d50\") " pod="openshift-image-registry/node-ca-wcchl"
Jul 15 20:47:49.794205 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.794042    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1bbd7d75-7a89-403d-983b-6534ab13e786-etc-sysconfig\") pod \"tuned-7wpzz\" (UID: \"1bbd7d75-7a89-403d-983b-6534ab13e786\") " pod="openshift-cluster-node-tuning-operator/tuned-7wpzz"
Jul 15 20:47:49.794205 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.794080    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1bbd7d75-7a89-403d-983b-6534ab13e786-etc-sysctl-d\") pod \"tuned-7wpzz\" (UID: \"1bbd7d75-7a89-403d-983b-6534ab13e786\") " pod="openshift-cluster-node-tuning-operator/tuned-7wpzz"
Jul 15 20:47:49.794205 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.794119    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1bbd7d75-7a89-403d-983b-6534ab13e786-etc-sysctl-conf\") pod \"tuned-7wpzz\" (UID: \"1bbd7d75-7a89-403d-983b-6534ab13e786\") " pod="openshift-cluster-node-tuning-operator/tuned-7wpzz"
Jul 15 20:47:49.794205 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.794143    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f9a159bb-39c4-4cca-92ff-1c74d739e311-host-slash\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.794205 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.794186    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f9a159bb-39c4-4cca-92ff-1c74d739e311-node-log\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.794205 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.794185    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1bbd7d75-7a89-403d-983b-6534ab13e786-etc-sysconfig\") pod \"tuned-7wpzz\" (UID: \"1bbd7d75-7a89-403d-983b-6534ab13e786\") " pod="openshift-cluster-node-tuning-operator/tuned-7wpzz"
Jul 15 20:47:49.794481 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.794250    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1bbd7d75-7a89-403d-983b-6534ab13e786-etc-modprobe-d\") pod \"tuned-7wpzz\" (UID: \"1bbd7d75-7a89-403d-983b-6534ab13e786\") " pod="openshift-cluster-node-tuning-operator/tuned-7wpzz"
Jul 15 20:47:49.794481 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.794275    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1bbd7d75-7a89-403d-983b-6534ab13e786-sys\") pod \"tuned-7wpzz\" (UID: \"1bbd7d75-7a89-403d-983b-6534ab13e786\") " pod="openshift-cluster-node-tuning-operator/tuned-7wpzz"
Jul 15 20:47:49.794481 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.794310    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e9ca36a8-de2e-472f-8b76-e6c4433f3ce4-socket-dir\") pod \"aws-ebs-csi-driver-node-8twzl\" (UID: \"e9ca36a8-de2e-472f-8b76-e6c4433f3ce4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8twzl"
Jul 15 20:47:49.794481 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.794345    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1bbd7d75-7a89-403d-983b-6534ab13e786-etc-sysctl-conf\") pod \"tuned-7wpzz\" (UID: \"1bbd7d75-7a89-403d-983b-6534ab13e786\") " pod="openshift-cluster-node-tuning-operator/tuned-7wpzz"
Jul 15 20:47:49.794481 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.794353    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1bbd7d75-7a89-403d-983b-6534ab13e786-etc-sysctl-d\") pod \"tuned-7wpzz\" (UID: \"1bbd7d75-7a89-403d-983b-6534ab13e786\") " pod="openshift-cluster-node-tuning-operator/tuned-7wpzz"
Jul 15 20:47:49.794713 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.794352    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j84nq\" (UniqueName: \"kubernetes.io/projected/e9ca36a8-de2e-472f-8b76-e6c4433f3ce4-kube-api-access-j84nq\") pod \"aws-ebs-csi-driver-node-8twzl\" (UID: \"e9ca36a8-de2e-472f-8b76-e6c4433f3ce4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8twzl"
Jul 15 20:47:49.794801 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.794767    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9a159bb-39c4-4cca-92ff-1c74d739e311-run-openvswitch\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.795549 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.795125    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f9a159bb-39c4-4cca-92ff-1c74d739e311-host-slash\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.795549 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.795204    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f9a159bb-39c4-4cca-92ff-1c74d739e311-node-log\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.795549 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.795223    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1bbd7d75-7a89-403d-983b-6534ab13e786-sys\") pod \"tuned-7wpzz\" (UID: \"1bbd7d75-7a89-403d-983b-6534ab13e786\") " pod="openshift-cluster-node-tuning-operator/tuned-7wpzz"
Jul 15 20:47:49.795549 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.795250    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1bbd7d75-7a89-403d-983b-6534ab13e786-etc-modprobe-d\") pod \"tuned-7wpzz\" (UID: \"1bbd7d75-7a89-403d-983b-6534ab13e786\") " pod="openshift-cluster-node-tuning-operator/tuned-7wpzz"
Jul 15 20:47:49.795549 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.795373    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9a159bb-39c4-4cca-92ff-1c74d739e311-run-openvswitch\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.795549 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.795456    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3166ef5d-e008-44a0-90e4-4f9c39f2332d-iptables-alerter-script\") pod \"iptables-alerter-n2vtn\" (UID: \"3166ef5d-e008-44a0-90e4-4f9c39f2332d\") " pod="openshift-network-operator/iptables-alerter-n2vtn"
Jul 15 20:47:49.795549 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.795529    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e9ca36a8-de2e-472f-8b76-e6c4433f3ce4-socket-dir\") pod \"aws-ebs-csi-driver-node-8twzl\" (UID: \"e9ca36a8-de2e-472f-8b76-e6c4433f3ce4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8twzl"
Jul 15 20:47:49.795983 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.795619    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1bfac382-72fc-400b-ab8b-9e4f53ab1d50-serviceca\") pod \"node-ca-wcchl\" (UID: \"1bfac382-72fc-400b-ab8b-9e4f53ab1d50\") " pod="openshift-image-registry/node-ca-wcchl"
Jul 15 20:47:49.795983 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.795642    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3166ef5d-e008-44a0-90e4-4f9c39f2332d-iptables-alerter-script\") pod \"iptables-alerter-n2vtn\" (UID: \"3166ef5d-e008-44a0-90e4-4f9c39f2332d\") " pod="openshift-network-operator/iptables-alerter-n2vtn"
Jul 15 20:47:49.795983 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.795702    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9a159bb-39c4-4cca-92ff-1c74d739e311-host-kubelet\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.795983 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.795751    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9a159bb-39c4-4cca-92ff-1c74d739e311-var-lib-openvswitch\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.795983 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.795785    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f9a159bb-39c4-4cca-92ff-1c74d739e311-env-overrides\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.795983 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.795786    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9a159bb-39c4-4cca-92ff-1c74d739e311-host-kubelet\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.795983 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.795822    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpb6j\" (UniqueName: \"kubernetes.io/projected/55baaf18-8e1a-4d1b-9411-ed55205cc56c-kube-api-access-gpb6j\") pod \"network-check-target-ngpfr\" (UID: \"55baaf18-8e1a-4d1b-9411-ed55205cc56c\") " pod="openshift-network-diagnostics/network-check-target-ngpfr"
Jul 15 20:47:49.795983 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.795849    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bbd7d75-7a89-403d-983b-6534ab13e786-etc-kubernetes\") pod \"tuned-7wpzz\" (UID: \"1bbd7d75-7a89-403d-983b-6534ab13e786\") " pod="openshift-cluster-node-tuning-operator/tuned-7wpzz"
Jul 15 20:47:49.795983 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.795868    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9a159bb-39c4-4cca-92ff-1c74d739e311-var-lib-openvswitch\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.795983 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.795882    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1bbd7d75-7a89-403d-983b-6534ab13e786-run\") pod \"tuned-7wpzz\" (UID: \"1bbd7d75-7a89-403d-983b-6534ab13e786\") " pod="openshift-cluster-node-tuning-operator/tuned-7wpzz"
Jul 15 20:47:49.795983 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.795961    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1bbd7d75-7a89-403d-983b-6534ab13e786-run\") pod \"tuned-7wpzz\" (UID: \"1bbd7d75-7a89-403d-983b-6534ab13e786\") " pod="openshift-cluster-node-tuning-operator/tuned-7wpzz"
Jul 15 20:47:49.796548 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.796123    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1bbd7d75-7a89-403d-983b-6534ab13e786-tmp\") pod \"tuned-7wpzz\" (UID: \"1bbd7d75-7a89-403d-983b-6534ab13e786\") " pod="openshift-cluster-node-tuning-operator/tuned-7wpzz"
Jul 15 20:47:49.796548 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.796028    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bbd7d75-7a89-403d-983b-6534ab13e786-etc-kubernetes\") pod \"tuned-7wpzz\" (UID: \"1bbd7d75-7a89-403d-983b-6534ab13e786\") " pod="openshift-cluster-node-tuning-operator/tuned-7wpzz"
Jul 15 20:47:49.796676 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.796617    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f9a159bb-39c4-4cca-92ff-1c74d739e311-run-systemd\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.796676 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.796643    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f9a159bb-39c4-4cca-92ff-1c74d739e311-env-overrides\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.796777 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.796674    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1bbd7d75-7a89-403d-983b-6534ab13e786-lib-modules\") pod \"tuned-7wpzz\" (UID: \"1bbd7d75-7a89-403d-983b-6534ab13e786\") " pod="openshift-cluster-node-tuning-operator/tuned-7wpzz"
Jul 15 20:47:49.796777 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.796712    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bbd7d75-7a89-403d-983b-6534ab13e786-var-lib-kubelet\") pod \"tuned-7wpzz\" (UID: \"1bbd7d75-7a89-403d-983b-6534ab13e786\") " pod="openshift-cluster-node-tuning-operator/tuned-7wpzz"
Jul 15 20:47:49.796777 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.796720    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f9a159bb-39c4-4cca-92ff-1c74d739e311-run-systemd\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.796777 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.796763    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9a159bb-39c4-4cca-92ff-1c74d739e311-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.796968 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.796788    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bbd7d75-7a89-403d-983b-6534ab13e786-var-lib-kubelet\") pod \"tuned-7wpzz\" (UID: \"1bbd7d75-7a89-403d-983b-6534ab13e786\") " pod="openshift-cluster-node-tuning-operator/tuned-7wpzz"
Jul 15 20:47:49.796968 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.796792    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f9a159bb-39c4-4cca-92ff-1c74d739e311-ovnkube-config\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.796968 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.796837    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f9a159bb-39c4-4cca-92ff-1c74d739e311-ovn-node-metrics-cert\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.796968 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.796882    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fpps\" (UniqueName: \"kubernetes.io/projected/3166ef5d-e008-44a0-90e4-4f9c39f2332d-kube-api-access-4fpps\") pod \"iptables-alerter-n2vtn\" (UID: \"3166ef5d-e008-44a0-90e4-4f9c39f2332d\") " pod="openshift-network-operator/iptables-alerter-n2vtn"
Jul 15 20:47:49.796968 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.796917    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e9ca36a8-de2e-472f-8b76-e6c4433f3ce4-registration-dir\") pod \"aws-ebs-csi-driver-node-8twzl\" (UID: \"e9ca36a8-de2e-472f-8b76-e6c4433f3ce4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8twzl"
Jul 15 20:47:49.796968 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.796967    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3166ef5d-e008-44a0-90e4-4f9c39f2332d-host-slash\") pod \"iptables-alerter-n2vtn\" (UID: \"3166ef5d-e008-44a0-90e4-4f9c39f2332d\") " pod="openshift-network-operator/iptables-alerter-n2vtn"
Jul 15 20:47:49.797260 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.797000    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e9ca36a8-de2e-472f-8b76-e6c4433f3ce4-device-dir\") pod \"aws-ebs-csi-driver-node-8twzl\" (UID: \"e9ca36a8-de2e-472f-8b76-e6c4433f3ce4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8twzl"
Jul 15 20:47:49.797260 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.797041    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1bbd7d75-7a89-403d-983b-6534ab13e786-etc-tuned\") pod \"tuned-7wpzz\" (UID: \"1bbd7d75-7a89-403d-983b-6534ab13e786\") " pod="openshift-cluster-node-tuning-operator/tuned-7wpzz"
Jul 15 20:47:49.797260 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.797068    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f9a159bb-39c4-4cca-92ff-1c74d739e311-systemd-units\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.797260 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.797098    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9a159bb-39c4-4cca-92ff-1c74d739e311-host-run-ovn-kubernetes\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.797260 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.797118    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e9ca36a8-de2e-472f-8b76-e6c4433f3ce4-etc-selinux\") pod \"aws-ebs-csi-driver-node-8twzl\" (UID: \"e9ca36a8-de2e-472f-8b76-e6c4433f3ce4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8twzl"
Jul 15 20:47:49.797260 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.797164    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f9a159bb-39c4-4cca-92ff-1c74d739e311-host-cni-bin\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.797260 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.797196    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e9ca36a8-de2e-472f-8b76-e6c4433f3ce4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8twzl\" (UID: \"e9ca36a8-de2e-472f-8b76-e6c4433f3ce4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8twzl"
Jul 15 20:47:49.797260 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.797217    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1bbd7d75-7a89-403d-983b-6534ab13e786-etc-systemd\") pod \"tuned-7wpzz\" (UID: \"1bbd7d75-7a89-403d-983b-6534ab13e786\") " pod="openshift-cluster-node-tuning-operator/tuned-7wpzz"
Jul 15 20:47:49.797260 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.797235    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1bbd7d75-7a89-403d-983b-6534ab13e786-host\") pod \"tuned-7wpzz\" (UID: \"1bbd7d75-7a89-403d-983b-6534ab13e786\") " pod="openshift-cluster-node-tuning-operator/tuned-7wpzz"
Jul 15 20:47:49.797687 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.797261    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrng5\" (UniqueName: \"kubernetes.io/projected/1bbd7d75-7a89-403d-983b-6534ab13e786-kube-api-access-nrng5\") pod \"tuned-7wpzz\" (UID: \"1bbd7d75-7a89-403d-983b-6534ab13e786\") " pod="openshift-cluster-node-tuning-operator/tuned-7wpzz"
Jul 15 20:47:49.797687 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.797266    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f9a159bb-39c4-4cca-92ff-1c74d739e311-ovnkube-config\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.797687 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.797286    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f9a159bb-39c4-4cca-92ff-1c74d739e311-host-cni-netd\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.797687 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.797321    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1bfac382-72fc-400b-ab8b-9e4f53ab1d50-host\") pod \"node-ca-wcchl\" (UID: \"1bfac382-72fc-400b-ab8b-9e4f53ab1d50\") " pod="openshift-image-registry/node-ca-wcchl"
Jul 15 20:47:49.797687 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.797338    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9a159bb-39c4-4cca-92ff-1c74d739e311-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.797687 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.797354    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e9ca36a8-de2e-472f-8b76-e6c4433f3ce4-sys-fs\") pod \"aws-ebs-csi-driver-node-8twzl\" (UID: \"e9ca36a8-de2e-472f-8b76-e6c4433f3ce4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8twzl"
Jul 15 20:47:49.797687 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.797379    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f9a159bb-39c4-4cca-92ff-1c74d739e311-host-run-netns\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.797687 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.797400    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f9a159bb-39c4-4cca-92ff-1c74d739e311-log-socket\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.797687 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.797427    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f9a159bb-39c4-4cca-92ff-1c74d739e311-ovnkube-script-lib\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.797687 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.797448    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pbmw\" (UniqueName: \"kubernetes.io/projected/1bfac382-72fc-400b-ab8b-9e4f53ab1d50-kube-api-access-4pbmw\") pod \"node-ca-wcchl\" (UID: \"1bfac382-72fc-400b-ab8b-9e4f53ab1d50\") " pod="openshift-image-registry/node-ca-wcchl"
Jul 15 20:47:49.797687 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.797471    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9a159bb-39c4-4cca-92ff-1c74d739e311-etc-openvswitch\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.797687 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.797503    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c252p\" (UniqueName: \"kubernetes.io/projected/f9a159bb-39c4-4cca-92ff-1c74d739e311-kube-api-access-c252p\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.797687 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.797540    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f9a159bb-39c4-4cca-92ff-1c74d739e311-run-ovn\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.797687 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.797633    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f9a159bb-39c4-4cca-92ff-1c74d739e311-run-ovn\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.798400 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.797760    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e9ca36a8-de2e-472f-8b76-e6c4433f3ce4-etc-selinux\") pod \"aws-ebs-csi-driver-node-8twzl\" (UID: \"e9ca36a8-de2e-472f-8b76-e6c4433f3ce4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8twzl"
Jul 15 20:47:49.798400 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.797820    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f9a159bb-39c4-4cca-92ff-1c74d739e311-host-cni-bin\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.798400 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.797883    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e9ca36a8-de2e-472f-8b76-e6c4433f3ce4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8twzl\" (UID: \"e9ca36a8-de2e-472f-8b76-e6c4433f3ce4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8twzl"
Jul 15 20:47:49.798400 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.797949    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1bbd7d75-7a89-403d-983b-6534ab13e786-etc-systemd\") pod \"tuned-7wpzz\" (UID: \"1bbd7d75-7a89-403d-983b-6534ab13e786\") " pod="openshift-cluster-node-tuning-operator/tuned-7wpzz"
Jul 15 20:47:49.798400 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.798032    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1bbd7d75-7a89-403d-983b-6534ab13e786-host\") pod \"tuned-7wpzz\" (UID: \"1bbd7d75-7a89-403d-983b-6534ab13e786\") " pod="openshift-cluster-node-tuning-operator/tuned-7wpzz"
Jul 15 20:47:49.798400 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.798340    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f9a159bb-39c4-4cca-92ff-1c74d739e311-host-cni-netd\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.798685 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.798405    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1bfac382-72fc-400b-ab8b-9e4f53ab1d50-host\") pod \"node-ca-wcchl\" (UID: \"1bfac382-72fc-400b-ab8b-9e4f53ab1d50\") " pod="openshift-image-registry/node-ca-wcchl"
Jul 15 20:47:49.798685 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.798482    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e9ca36a8-de2e-472f-8b76-e6c4433f3ce4-sys-fs\") pod \"aws-ebs-csi-driver-node-8twzl\" (UID: \"e9ca36a8-de2e-472f-8b76-e6c4433f3ce4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8twzl"
Jul 15 20:47:49.798685 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.798532    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9a159bb-39c4-4cca-92ff-1c74d739e311-host-run-ovn-kubernetes\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.798685 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.798643    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1bbd7d75-7a89-403d-983b-6534ab13e786-lib-modules\") pod \"tuned-7wpzz\" (UID: \"1bbd7d75-7a89-403d-983b-6534ab13e786\") " pod="openshift-cluster-node-tuning-operator/tuned-7wpzz"
Jul 15 20:47:49.798888 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.798700    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f9a159bb-39c4-4cca-92ff-1c74d739e311-host-run-netns\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.798888 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.798749    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f9a159bb-39c4-4cca-92ff-1c74d739e311-log-socket\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.799418 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.799070    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3166ef5d-e008-44a0-90e4-4f9c39f2332d-host-slash\") pod \"iptables-alerter-n2vtn\" (UID: \"3166ef5d-e008-44a0-90e4-4f9c39f2332d\") " pod="openshift-network-operator/iptables-alerter-n2vtn"
Jul 15 20:47:49.799418 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.799218    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1bbd7d75-7a89-403d-983b-6534ab13e786-tmp\") pod \"tuned-7wpzz\" (UID: \"1bbd7d75-7a89-403d-983b-6534ab13e786\") " pod="openshift-cluster-node-tuning-operator/tuned-7wpzz"
Jul 15 20:47:49.799418 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.799303    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e9ca36a8-de2e-472f-8b76-e6c4433f3ce4-registration-dir\") pod \"aws-ebs-csi-driver-node-8twzl\" (UID: \"e9ca36a8-de2e-472f-8b76-e6c4433f3ce4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8twzl"
Jul 15 20:47:49.799418 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.799334    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f9a159bb-39c4-4cca-92ff-1c74d739e311-ovnkube-script-lib\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.799418 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.799366    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9a159bb-39c4-4cca-92ff-1c74d739e311-etc-openvswitch\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.799724 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.799429    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f9a159bb-39c4-4cca-92ff-1c74d739e311-systemd-units\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.799724 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.799707    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e9ca36a8-de2e-472f-8b76-e6c4433f3ce4-device-dir\") pod \"aws-ebs-csi-driver-node-8twzl\" (UID: \"e9ca36a8-de2e-472f-8b76-e6c4433f3ce4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8twzl"
Jul 15 20:47:49.800269 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.799915    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f9a159bb-39c4-4cca-92ff-1c74d739e311-ovn-node-metrics-cert\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.802473 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.802430    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1bbd7d75-7a89-403d-983b-6534ab13e786-etc-tuned\") pod \"tuned-7wpzz\" (UID: \"1bbd7d75-7a89-403d-983b-6534ab13e786\") " pod="openshift-cluster-node-tuning-operator/tuned-7wpzz"
Jul 15 20:47:49.804402 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:49.804379    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 15 20:47:49.804402 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:49.804405    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 15 20:47:49.804569 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:49.804417    2566 projected.go:196] Error preparing data for projected volume kube-api-access-gpb6j for pod openshift-network-diagnostics/network-check-target-ngpfr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 20:47:49.804569 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:49.804466    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55baaf18-8e1a-4d1b-9411-ed55205cc56c-kube-api-access-gpb6j podName:55baaf18-8e1a-4d1b-9411-ed55205cc56c nodeName:}" failed. No retries permitted until 2026-07-15 20:47:50.304452738 +0000 UTC m=+3.453490792 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gpb6j" (UniqueName: "kubernetes.io/projected/55baaf18-8e1a-4d1b-9411-ed55205cc56c-kube-api-access-gpb6j") pod "network-check-target-ngpfr" (UID: "55baaf18-8e1a-4d1b-9411-ed55205cc56c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 20:47:49.806899 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.806875    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j84nq\" (UniqueName: \"kubernetes.io/projected/e9ca36a8-de2e-472f-8b76-e6c4433f3ce4-kube-api-access-j84nq\") pod \"aws-ebs-csi-driver-node-8twzl\" (UID: \"e9ca36a8-de2e-472f-8b76-e6c4433f3ce4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8twzl"
Jul 15 20:47:49.807115 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.807096    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrng5\" (UniqueName: \"kubernetes.io/projected/1bbd7d75-7a89-403d-983b-6534ab13e786-kube-api-access-nrng5\") pod \"tuned-7wpzz\" (UID: \"1bbd7d75-7a89-403d-983b-6534ab13e786\") " pod="openshift-cluster-node-tuning-operator/tuned-7wpzz"
Jul 15 20:47:49.811331 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.811307    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fpps\" (UniqueName: \"kubernetes.io/projected/3166ef5d-e008-44a0-90e4-4f9c39f2332d-kube-api-access-4fpps\") pod \"iptables-alerter-n2vtn\" (UID: \"3166ef5d-e008-44a0-90e4-4f9c39f2332d\") " pod="openshift-network-operator/iptables-alerter-n2vtn"
Jul 15 20:47:49.821631 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.821606    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c252p\" (UniqueName: \"kubernetes.io/projected/f9a159bb-39c4-4cca-92ff-1c74d739e311-kube-api-access-c252p\") pod \"ovnkube-node-652lk\" (UID: \"f9a159bb-39c4-4cca-92ff-1c74d739e311\") " pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:49.821770 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.821746    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pbmw\" (UniqueName: \"kubernetes.io/projected/1bfac382-72fc-400b-ab8b-9e4f53ab1d50-kube-api-access-4pbmw\") pod \"node-ca-wcchl\" (UID: \"1bfac382-72fc-400b-ab8b-9e4f53ab1d50\") " pod="openshift-image-registry/node-ca-wcchl"
Jul 15 20:47:49.879011 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.878971    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wjxpz"
Jul 15 20:47:49.889090 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.889061    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qvvx2"
Jul 15 20:47:49.900480 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.900342    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6j89c"
Jul 15 20:47:49.907791 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.907771    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wvh9c"
Jul 15 20:47:49.915460 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.915442    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wcchl"
Jul 15 20:47:49.927166 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.927114    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-n2vtn"
Jul 15 20:47:49.933199 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.933179    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8twzl"
Jul 15 20:47:49.943048 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.943022    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7wpzz"
Jul 15 20:47:49.948889 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:49.948870    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:47:50.201110 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:50.201014    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9d9267e-0c23-42b2-bb48-55ed14050fff-original-pull-secret\") pod \"global-pull-secret-syncer-zzw8h\" (UID: \"b9d9267e-0c23-42b2-bb48-55ed14050fff\") " pod="kube-system/global-pull-secret-syncer-zzw8h"
Jul 15 20:47:50.201110 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:50.201084    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42a42d42-83b0-441b-b7f2-3770f7523317-metrics-certs\") pod \"network-metrics-daemon-xwbql\" (UID: \"42a42d42-83b0-441b-b7f2-3770f7523317\") " pod="openshift-multus/network-metrics-daemon-xwbql"
Jul 15 20:47:50.201336 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:50.201209    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 20:47:50.201336 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:50.201274    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 15 20:47:50.201443 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:50.201282    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42a42d42-83b0-441b-b7f2-3770f7523317-metrics-certs podName:42a42d42-83b0-441b-b7f2-3770f7523317 nodeName:}" failed. No retries permitted until 2026-07-15 20:47:51.201263402 +0000 UTC m=+4.350301445 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/42a42d42-83b0-441b-b7f2-3770f7523317-metrics-certs") pod "network-metrics-daemon-xwbql" (UID: "42a42d42-83b0-441b-b7f2-3770f7523317") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 20:47:50.201443 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:50.201375    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9d9267e-0c23-42b2-bb48-55ed14050fff-original-pull-secret podName:b9d9267e-0c23-42b2-bb48-55ed14050fff nodeName:}" failed. No retries permitted until 2026-07-15 20:47:51.201352047 +0000 UTC m=+4.350390102 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b9d9267e-0c23-42b2-bb48-55ed14050fff-original-pull-secret") pod "global-pull-secret-syncer-zzw8h" (UID: "b9d9267e-0c23-42b2-bb48-55ed14050fff") : object "kube-system"/"original-pull-secret" not registered
Jul 15 20:47:50.403140 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:50.403099    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpb6j\" (UniqueName: \"kubernetes.io/projected/55baaf18-8e1a-4d1b-9411-ed55205cc56c-kube-api-access-gpb6j\") pod \"network-check-target-ngpfr\" (UID: \"55baaf18-8e1a-4d1b-9411-ed55205cc56c\") " pod="openshift-network-diagnostics/network-check-target-ngpfr"
Jul 15 20:47:50.403337 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:50.403254    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 15 20:47:50.403337 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:50.403277    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 15 20:47:50.403337 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:50.403292    2566 projected.go:196] Error preparing data for projected volume kube-api-access-gpb6j for pod openshift-network-diagnostics/network-check-target-ngpfr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 20:47:50.403482 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:50.403353    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55baaf18-8e1a-4d1b-9411-ed55205cc56c-kube-api-access-gpb6j podName:55baaf18-8e1a-4d1b-9411-ed55205cc56c nodeName:}" failed. No retries permitted until 2026-07-15 20:47:51.403337119 +0000 UTC m=+4.552375156 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gpb6j" (UniqueName: "kubernetes.io/projected/55baaf18-8e1a-4d1b-9411-ed55205cc56c-kube-api-access-gpb6j") pod "network-check-target-ngpfr" (UID: "55baaf18-8e1a-4d1b-9411-ed55205cc56c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 20:47:50.453414 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:50.453331    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 15 20:47:50.642089 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:50.642034    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 20:42:49 +0000 UTC" deadline="2028-02-23 06:34:58.24097767 +0000 UTC"
Jul 15 20:47:50.642089 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:50.642077    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14097h47m7.598906099s"
Jul 15 20:47:50.905939 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:50.905906    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bbd7d75_7a89_403d_983b_6534ab13e786.slice/crio-69996afeac291e6347fd62ef24437fbb3eeee1b481ce6285d0d2badc4dc2d161 WatchSource:0}: Error finding container 69996afeac291e6347fd62ef24437fbb3eeee1b481ce6285d0d2badc4dc2d161: Status 404 returned error can't find the container with id 69996afeac291e6347fd62ef24437fbb3eeee1b481ce6285d0d2badc4dc2d161
Jul 15 20:47:50.910187 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:50.910044    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod147d4538_e5f1_47af_b61c_f07de8e0b0df.slice/crio-9a3ccdfacfed5dc0edaf9dcc1e09b0bebe43b56b603cb6a93cf46ea2c104de09 WatchSource:0}: Error finding container 9a3ccdfacfed5dc0edaf9dcc1e09b0bebe43b56b603cb6a93cf46ea2c104de09: Status 404 returned error can't find the container with id 9a3ccdfacfed5dc0edaf9dcc1e09b0bebe43b56b603cb6a93cf46ea2c104de09
Jul 15 20:47:50.912708 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:50.912683    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9ca36a8_de2e_472f_8b76_e6c4433f3ce4.slice/crio-6fa80723212a84f84a1a39be83c3948aebd6ae9fc865a509c50b59c41fc0ab86 WatchSource:0}: Error finding container 6fa80723212a84f84a1a39be83c3948aebd6ae9fc865a509c50b59c41fc0ab86: Status 404 returned error can't find the container with id 6fa80723212a84f84a1a39be83c3948aebd6ae9fc865a509c50b59c41fc0ab86
Jul 15 20:47:50.913597 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:50.913574    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bfac382_72fc_400b_ab8b_9e4f53ab1d50.slice/crio-c5495bc80133ad7711a36e8f3864594f42e4d57608ee7c8f38fe591050240d44 WatchSource:0}: Error finding container c5495bc80133ad7711a36e8f3864594f42e4d57608ee7c8f38fe591050240d44: Status 404 returned error can't find the container with id c5495bc80133ad7711a36e8f3864594f42e4d57608ee7c8f38fe591050240d44
Jul 15 20:47:50.915619 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:50.915158    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dbf2724_61f8_4572_a2c1_906032db929d.slice/crio-47ab6147d9a9ed6a81512f0a88c642e7cfc0463e6c4fd29901e548eca6172f4c WatchSource:0}: Error finding container 47ab6147d9a9ed6a81512f0a88c642e7cfc0463e6c4fd29901e548eca6172f4c: Status 404 returned error can't find the container with id 47ab6147d9a9ed6a81512f0a88c642e7cfc0463e6c4fd29901e548eca6172f4c
Jul 15 20:47:50.920764 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:50.920299    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9a159bb_39c4_4cca_92ff_1c74d739e311.slice/crio-2d1d59b60c030f352f4ce00f74a5d8b556d72ca3d5823aa842f10898b988c17f WatchSource:0}: Error finding container 2d1d59b60c030f352f4ce00f74a5d8b556d72ca3d5823aa842f10898b988c17f: Status 404 returned error can't find the container with id 2d1d59b60c030f352f4ce00f74a5d8b556d72ca3d5823aa842f10898b988c17f
Jul 15 20:47:50.922433 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:50.921557    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad23d165_aeec_409f_99bd_12c27029221b.slice/crio-529a6d3830f597780e1f818eae7a67af953a44f6ff56b47c33a10b96e0ba06c2 WatchSource:0}: Error finding container 529a6d3830f597780e1f818eae7a67af953a44f6ff56b47c33a10b96e0ba06c2: Status 404 returned error can't find the container with id 529a6d3830f597780e1f818eae7a67af953a44f6ff56b47c33a10b96e0ba06c2
Jul 15 20:47:50.922433 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:47:50.922083    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3166ef5d_e008_44a0_90e4_4f9c39f2332d.slice/crio-574540b85b9bf40a3a560ddf80df733a72e5808d317ee34bf00c87c2602bdfdb WatchSource:0}: Error finding container 574540b85b9bf40a3a560ddf80df733a72e5808d317ee34bf00c87c2602bdfdb: Status 404 returned error can't find the container with id 574540b85b9bf40a3a560ddf80df733a72e5808d317ee34bf00c87c2602bdfdb
Jul 15 20:47:51.209093 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:51.208912    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42a42d42-83b0-441b-b7f2-3770f7523317-metrics-certs\") pod \"network-metrics-daemon-xwbql\" (UID: \"42a42d42-83b0-441b-b7f2-3770f7523317\") " pod="openshift-multus/network-metrics-daemon-xwbql"
Jul 15 20:47:51.209241 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:51.209132    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9d9267e-0c23-42b2-bb48-55ed14050fff-original-pull-secret\") pod \"global-pull-secret-syncer-zzw8h\" (UID: \"b9d9267e-0c23-42b2-bb48-55ed14050fff\") " pod="kube-system/global-pull-secret-syncer-zzw8h"
Jul 15 20:47:51.209241 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:51.208996    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 20:47:51.209241 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:51.209205    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42a42d42-83b0-441b-b7f2-3770f7523317-metrics-certs podName:42a42d42-83b0-441b-b7f2-3770f7523317 nodeName:}" failed. No retries permitted until 2026-07-15 20:47:53.20918955 +0000 UTC m=+6.358227585 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/42a42d42-83b0-441b-b7f2-3770f7523317-metrics-certs") pod "network-metrics-daemon-xwbql" (UID: "42a42d42-83b0-441b-b7f2-3770f7523317") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 20:47:51.209365 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:51.209262    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 15 20:47:51.209365 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:51.209310    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9d9267e-0c23-42b2-bb48-55ed14050fff-original-pull-secret podName:b9d9267e-0c23-42b2-bb48-55ed14050fff nodeName:}" failed. No retries permitted until 2026-07-15 20:47:53.209296245 +0000 UTC m=+6.358334281 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b9d9267e-0c23-42b2-bb48-55ed14050fff-original-pull-secret") pod "global-pull-secret-syncer-zzw8h" (UID: "b9d9267e-0c23-42b2-bb48-55ed14050fff") : object "kube-system"/"original-pull-secret" not registered
Jul 15 20:47:51.411390 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:51.411297    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpb6j\" (UniqueName: \"kubernetes.io/projected/55baaf18-8e1a-4d1b-9411-ed55205cc56c-kube-api-access-gpb6j\") pod \"network-check-target-ngpfr\" (UID: \"55baaf18-8e1a-4d1b-9411-ed55205cc56c\") " pod="openshift-network-diagnostics/network-check-target-ngpfr"
Jul 15 20:47:51.411549 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:51.411503    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 15 20:47:51.411549 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:51.411522    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 15 20:47:51.411549 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:51.411534    2566 projected.go:196] Error preparing data for projected volume kube-api-access-gpb6j for pod openshift-network-diagnostics/network-check-target-ngpfr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 20:47:51.411731 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:51.411600    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55baaf18-8e1a-4d1b-9411-ed55205cc56c-kube-api-access-gpb6j podName:55baaf18-8e1a-4d1b-9411-ed55205cc56c nodeName:}" failed. No retries permitted until 2026-07-15 20:47:53.411581621 +0000 UTC m=+6.560619679 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gpb6j" (UniqueName: "kubernetes.io/projected/55baaf18-8e1a-4d1b-9411-ed55205cc56c-kube-api-access-gpb6j") pod "network-check-target-ngpfr" (UID: "55baaf18-8e1a-4d1b-9411-ed55205cc56c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 20:47:51.642629 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:51.642545    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 20:42:49 +0000 UTC" deadline="2028-02-24 05:02:33.473000584 +0000 UTC"
Jul 15 20:47:51.642629 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:51.642601    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14120h14m41.830404294s"
Jul 15 20:47:51.751123 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:51.751047    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwbql"
Jul 15 20:47:51.751273 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:51.751052    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzw8h"
Jul 15 20:47:51.751273 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:51.751223    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwbql" podUID="42a42d42-83b0-441b-b7f2-3770f7523317"
Jul 15 20:47:51.751379 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:51.751323    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzw8h" podUID="b9d9267e-0c23-42b2-bb48-55ed14050fff"
Jul 15 20:47:51.751430 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:51.751384    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngpfr"
Jul 15 20:47:51.751483 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:51.751452    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngpfr" podUID="55baaf18-8e1a-4d1b-9411-ed55205cc56c"
Jul 15 20:47:51.766368 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:51.766334    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wcchl" event={"ID":"1bfac382-72fc-400b-ab8b-9e4f53ab1d50","Type":"ContainerStarted","Data":"c5495bc80133ad7711a36e8f3864594f42e4d57608ee7c8f38fe591050240d44"}
Jul 15 20:47:51.774864 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:51.774836    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8twzl" event={"ID":"e9ca36a8-de2e-472f-8b76-e6c4433f3ce4","Type":"ContainerStarted","Data":"6fa80723212a84f84a1a39be83c3948aebd6ae9fc865a509c50b59c41fc0ab86"}
Jul 15 20:47:51.783450 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:51.783422    2566 generic.go:356] "Generic (PLEG): container finished" podID="b740363c6967fb438e6610aa1d6ea6c6" containerID="c84aeec58e1f4c2a1149720c7abb648dd075b0cca5c05d3518a60cd9e6271b23" exitCode=0
Jul 15 20:47:51.783562 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:51.783500    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-234.ec2.internal" event={"ID":"b740363c6967fb438e6610aa1d6ea6c6","Type":"ContainerDied","Data":"c84aeec58e1f4c2a1149720c7abb648dd075b0cca5c05d3518a60cd9e6271b23"}
Jul 15 20:47:51.802601 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:51.802573    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-234.ec2.internal" event={"ID":"4a1659746931c9e82b46a4829b7de6fa","Type":"ContainerStarted","Data":"e56b2abf4b1e818592abe01ff6aa917121ade52b878ec563466141ca55e4bb0c"}
Jul 15 20:47:51.812629 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:51.812132    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-n2vtn" event={"ID":"3166ef5d-e008-44a0-90e4-4f9c39f2332d","Type":"ContainerStarted","Data":"574540b85b9bf40a3a560ddf80df733a72e5808d317ee34bf00c87c2602bdfdb"}
Jul 15 20:47:51.830019 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:51.828830    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wvh9c" event={"ID":"d786587a-4e93-443c-bfaa-334ced76d272","Type":"ContainerStarted","Data":"9da80bb1e710bc627a8be3ec05d2843628f2b6ba30e4554959e65fc608a18ae4"}
Jul 15 20:47:51.831510 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:51.831337    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-234.ec2.internal" podStartSLOduration=2.831319452 podStartE2EDuration="2.831319452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 20:47:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:47:51.82934595 +0000 UTC m=+4.978384008" watchObservedRunningTime="2026-07-15 20:47:51.831319452 +0000 UTC m=+4.980357511"
Jul 15 20:47:51.839876 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:51.839849    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qvvx2" event={"ID":"147d4538-e5f1-47af-b61c-f07de8e0b0df","Type":"ContainerStarted","Data":"9a3ccdfacfed5dc0edaf9dcc1e09b0bebe43b56b603cb6a93cf46ea2c104de09"}
Jul 15 20:47:51.844858 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:51.844831    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7wpzz" event={"ID":"1bbd7d75-7a89-403d-983b-6534ab13e786","Type":"ContainerStarted","Data":"69996afeac291e6347fd62ef24437fbb3eeee1b481ce6285d0d2badc4dc2d161"}
Jul 15 20:47:51.857108 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:51.857081    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6j89c" event={"ID":"ad23d165-aeec-409f-99bd-12c27029221b","Type":"ContainerStarted","Data":"529a6d3830f597780e1f818eae7a67af953a44f6ff56b47c33a10b96e0ba06c2"}
Jul 15 20:47:51.874839 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:51.874806    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-652lk" event={"ID":"f9a159bb-39c4-4cca-92ff-1c74d739e311","Type":"ContainerStarted","Data":"2d1d59b60c030f352f4ce00f74a5d8b556d72ca3d5823aa842f10898b988c17f"}
Jul 15 20:47:51.893559 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:51.893531    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wjxpz" event={"ID":"2dbf2724-61f8-4572-a2c1-906032db929d","Type":"ContainerStarted","Data":"47ab6147d9a9ed6a81512f0a88c642e7cfc0463e6c4fd29901e548eca6172f4c"}
Jul 15 20:47:52.921044 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:52.920972    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-234.ec2.internal" event={"ID":"b740363c6967fb438e6610aa1d6ea6c6","Type":"ContainerStarted","Data":"6f180772be566460a21a047f8acec0227014fdfba832f8310629abefdc1df707"}
Jul 15 20:47:53.227087 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:53.226336    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42a42d42-83b0-441b-b7f2-3770f7523317-metrics-certs\") pod \"network-metrics-daemon-xwbql\" (UID: \"42a42d42-83b0-441b-b7f2-3770f7523317\") " pod="openshift-multus/network-metrics-daemon-xwbql"
Jul 15 20:47:53.227087 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:53.226418    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9d9267e-0c23-42b2-bb48-55ed14050fff-original-pull-secret\") pod \"global-pull-secret-syncer-zzw8h\" (UID: \"b9d9267e-0c23-42b2-bb48-55ed14050fff\") " pod="kube-system/global-pull-secret-syncer-zzw8h"
Jul 15 20:47:53.227087 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:53.226543    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 15 20:47:53.227087 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:53.226605    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9d9267e-0c23-42b2-bb48-55ed14050fff-original-pull-secret podName:b9d9267e-0c23-42b2-bb48-55ed14050fff nodeName:}" failed. No retries permitted until 2026-07-15 20:47:57.226587777 +0000 UTC m=+10.375625829 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b9d9267e-0c23-42b2-bb48-55ed14050fff-original-pull-secret") pod "global-pull-secret-syncer-zzw8h" (UID: "b9d9267e-0c23-42b2-bb48-55ed14050fff") : object "kube-system"/"original-pull-secret" not registered
Jul 15 20:47:53.227087 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:53.227002    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 20:47:53.227087 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:53.227054    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42a42d42-83b0-441b-b7f2-3770f7523317-metrics-certs podName:42a42d42-83b0-441b-b7f2-3770f7523317 nodeName:}" failed. No retries permitted until 2026-07-15 20:47:57.227037141 +0000 UTC m=+10.376075178 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/42a42d42-83b0-441b-b7f2-3770f7523317-metrics-certs") pod "network-metrics-daemon-xwbql" (UID: "42a42d42-83b0-441b-b7f2-3770f7523317") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 20:47:53.428509 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:53.427790    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpb6j\" (UniqueName: \"kubernetes.io/projected/55baaf18-8e1a-4d1b-9411-ed55205cc56c-kube-api-access-gpb6j\") pod \"network-check-target-ngpfr\" (UID: \"55baaf18-8e1a-4d1b-9411-ed55205cc56c\") " pod="openshift-network-diagnostics/network-check-target-ngpfr"
Jul 15 20:47:53.428509 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:53.427993    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 15 20:47:53.428509 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:53.428013    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 15 20:47:53.428509 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:53.428045    2566 projected.go:196] Error preparing data for projected volume kube-api-access-gpb6j for pod openshift-network-diagnostics/network-check-target-ngpfr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 20:47:53.428509 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:53.428104    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55baaf18-8e1a-4d1b-9411-ed55205cc56c-kube-api-access-gpb6j podName:55baaf18-8e1a-4d1b-9411-ed55205cc56c nodeName:}" failed. No retries permitted until 2026-07-15 20:47:57.428085858 +0000 UTC m=+10.577123898 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gpb6j" (UniqueName: "kubernetes.io/projected/55baaf18-8e1a-4d1b-9411-ed55205cc56c-kube-api-access-gpb6j") pod "network-check-target-ngpfr" (UID: "55baaf18-8e1a-4d1b-9411-ed55205cc56c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 20:47:53.751072 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:53.751037    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwbql"
Jul 15 20:47:53.751276 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:53.751186    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwbql" podUID="42a42d42-83b0-441b-b7f2-3770f7523317"
Jul 15 20:47:53.751575 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:53.751554    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzw8h"
Jul 15 20:47:53.751755 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:53.751689    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzw8h" podUID="b9d9267e-0c23-42b2-bb48-55ed14050fff"
Jul 15 20:47:53.751755 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:53.751748    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngpfr"
Jul 15 20:47:53.751925 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:53.751816    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngpfr" podUID="55baaf18-8e1a-4d1b-9411-ed55205cc56c"
Jul 15 20:47:55.752705 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:55.752008    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzw8h"
Jul 15 20:47:55.752705 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:55.752140    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzw8h" podUID="b9d9267e-0c23-42b2-bb48-55ed14050fff"
Jul 15 20:47:55.752705 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:55.752560    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngpfr"
Jul 15 20:47:55.752705 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:55.752654    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngpfr" podUID="55baaf18-8e1a-4d1b-9411-ed55205cc56c"
Jul 15 20:47:55.753572 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:55.753425    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwbql"
Jul 15 20:47:55.753572 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:55.753536    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwbql" podUID="42a42d42-83b0-441b-b7f2-3770f7523317"
Jul 15 20:47:57.261769 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:57.261606    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9d9267e-0c23-42b2-bb48-55ed14050fff-original-pull-secret\") pod \"global-pull-secret-syncer-zzw8h\" (UID: \"b9d9267e-0c23-42b2-bb48-55ed14050fff\") " pod="kube-system/global-pull-secret-syncer-zzw8h"
Jul 15 20:47:57.261769 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:57.261693    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42a42d42-83b0-441b-b7f2-3770f7523317-metrics-certs\") pod \"network-metrics-daemon-xwbql\" (UID: \"42a42d42-83b0-441b-b7f2-3770f7523317\") " pod="openshift-multus/network-metrics-daemon-xwbql"
Jul 15 20:47:57.262426 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:57.261776    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 15 20:47:57.262426 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:57.261836    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 20:47:57.262426 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:57.261859    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9d9267e-0c23-42b2-bb48-55ed14050fff-original-pull-secret podName:b9d9267e-0c23-42b2-bb48-55ed14050fff nodeName:}" failed. No retries permitted until 2026-07-15 20:48:05.261837759 +0000 UTC m=+18.410875814 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b9d9267e-0c23-42b2-bb48-55ed14050fff-original-pull-secret") pod "global-pull-secret-syncer-zzw8h" (UID: "b9d9267e-0c23-42b2-bb48-55ed14050fff") : object "kube-system"/"original-pull-secret" not registered
Jul 15 20:47:57.262426 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:57.261921    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42a42d42-83b0-441b-b7f2-3770f7523317-metrics-certs podName:42a42d42-83b0-441b-b7f2-3770f7523317 nodeName:}" failed. No retries permitted until 2026-07-15 20:48:05.261888588 +0000 UTC m=+18.410926646 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/42a42d42-83b0-441b-b7f2-3770f7523317-metrics-certs") pod "network-metrics-daemon-xwbql" (UID: "42a42d42-83b0-441b-b7f2-3770f7523317") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 20:47:57.463026 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:57.462988    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpb6j\" (UniqueName: \"kubernetes.io/projected/55baaf18-8e1a-4d1b-9411-ed55205cc56c-kube-api-access-gpb6j\") pod \"network-check-target-ngpfr\" (UID: \"55baaf18-8e1a-4d1b-9411-ed55205cc56c\") " pod="openshift-network-diagnostics/network-check-target-ngpfr"
Jul 15 20:47:57.463210 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:57.463150    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 15 20:47:57.463210 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:57.463171    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 15 20:47:57.463210 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:57.463185    2566 projected.go:196] Error preparing data for projected volume kube-api-access-gpb6j for pod openshift-network-diagnostics/network-check-target-ngpfr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 20:47:57.463371 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:57.463251    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55baaf18-8e1a-4d1b-9411-ed55205cc56c-kube-api-access-gpb6j podName:55baaf18-8e1a-4d1b-9411-ed55205cc56c nodeName:}" failed. No retries permitted until 2026-07-15 20:48:05.463231304 +0000 UTC m=+18.612269339 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gpb6j" (UniqueName: "kubernetes.io/projected/55baaf18-8e1a-4d1b-9411-ed55205cc56c-kube-api-access-gpb6j") pod "network-check-target-ngpfr" (UID: "55baaf18-8e1a-4d1b-9411-ed55205cc56c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 20:47:57.751997 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:57.751963    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwbql"
Jul 15 20:47:57.752237 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:57.752092    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwbql" podUID="42a42d42-83b0-441b-b7f2-3770f7523317"
Jul 15 20:47:57.752237 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:57.752200    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzw8h"
Jul 15 20:47:57.752425 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:57.752372    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzw8h" podUID="b9d9267e-0c23-42b2-bb48-55ed14050fff"
Jul 15 20:47:57.752485 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:57.752455    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngpfr"
Jul 15 20:47:57.752567 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:57.752548    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngpfr" podUID="55baaf18-8e1a-4d1b-9411-ed55205cc56c"
Jul 15 20:47:59.752151 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:59.751454    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwbql"
Jul 15 20:47:59.752151 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:59.751595    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwbql" podUID="42a42d42-83b0-441b-b7f2-3770f7523317"
Jul 15 20:47:59.752151 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:59.752009    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzw8h"
Jul 15 20:47:59.752151 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:59.752103    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzw8h" podUID="b9d9267e-0c23-42b2-bb48-55ed14050fff"
Jul 15 20:47:59.752768 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:47:59.752247    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngpfr"
Jul 15 20:47:59.752768 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:47:59.752348    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngpfr" podUID="55baaf18-8e1a-4d1b-9411-ed55205cc56c"
Jul 15 20:48:01.751400 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:01.751354    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngpfr"
Jul 15 20:48:01.751400 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:01.751380    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwbql"
Jul 15 20:48:01.751400 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:01.751389    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzw8h"
Jul 15 20:48:01.751928 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:01.751488    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngpfr" podUID="55baaf18-8e1a-4d1b-9411-ed55205cc56c"
Jul 15 20:48:01.751928 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:01.751637    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwbql" podUID="42a42d42-83b0-441b-b7f2-3770f7523317"
Jul 15 20:48:01.751928 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:01.751704    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzw8h" podUID="b9d9267e-0c23-42b2-bb48-55ed14050fff"
Jul 15 20:48:03.751022 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:03.750985    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwbql"
Jul 15 20:48:03.751022 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:03.751009    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzw8h"
Jul 15 20:48:03.751022 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:03.751035    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngpfr"
Jul 15 20:48:03.751556 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:03.751131    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwbql" podUID="42a42d42-83b0-441b-b7f2-3770f7523317"
Jul 15 20:48:03.751556 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:03.751240    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzw8h" podUID="b9d9267e-0c23-42b2-bb48-55ed14050fff"
Jul 15 20:48:03.751556 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:03.751330    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngpfr" podUID="55baaf18-8e1a-4d1b-9411-ed55205cc56c"
Jul 15 20:48:05.315967 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:05.315925    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9d9267e-0c23-42b2-bb48-55ed14050fff-original-pull-secret\") pod \"global-pull-secret-syncer-zzw8h\" (UID: \"b9d9267e-0c23-42b2-bb48-55ed14050fff\") " pod="kube-system/global-pull-secret-syncer-zzw8h"
Jul 15 20:48:05.316367 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:05.315987    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42a42d42-83b0-441b-b7f2-3770f7523317-metrics-certs\") pod \"network-metrics-daemon-xwbql\" (UID: \"42a42d42-83b0-441b-b7f2-3770f7523317\") " pod="openshift-multus/network-metrics-daemon-xwbql"
Jul 15 20:48:05.316367 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:05.316082    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 15 20:48:05.316367 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:05.316148    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9d9267e-0c23-42b2-bb48-55ed14050fff-original-pull-secret podName:b9d9267e-0c23-42b2-bb48-55ed14050fff nodeName:}" failed. No retries permitted until 2026-07-15 20:48:21.316130495 +0000 UTC m=+34.465168534 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b9d9267e-0c23-42b2-bb48-55ed14050fff-original-pull-secret") pod "global-pull-secret-syncer-zzw8h" (UID: "b9d9267e-0c23-42b2-bb48-55ed14050fff") : object "kube-system"/"original-pull-secret" not registered
Jul 15 20:48:05.316367 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:05.316084    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 20:48:05.316367 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:05.316214    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42a42d42-83b0-441b-b7f2-3770f7523317-metrics-certs podName:42a42d42-83b0-441b-b7f2-3770f7523317 nodeName:}" failed. No retries permitted until 2026-07-15 20:48:21.316201405 +0000 UTC m=+34.465239443 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/42a42d42-83b0-441b-b7f2-3770f7523317-metrics-certs") pod "network-metrics-daemon-xwbql" (UID: "42a42d42-83b0-441b-b7f2-3770f7523317") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 20:48:05.517878 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:05.517835    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpb6j\" (UniqueName: \"kubernetes.io/projected/55baaf18-8e1a-4d1b-9411-ed55205cc56c-kube-api-access-gpb6j\") pod \"network-check-target-ngpfr\" (UID: \"55baaf18-8e1a-4d1b-9411-ed55205cc56c\") " pod="openshift-network-diagnostics/network-check-target-ngpfr"
Jul 15 20:48:05.518128 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:05.518027    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 15 20:48:05.518128 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:05.518055    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 15 20:48:05.518128 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:05.518067    2566 projected.go:196] Error preparing data for projected volume kube-api-access-gpb6j for pod openshift-network-diagnostics/network-check-target-ngpfr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 20:48:05.518128 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:05.518124    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55baaf18-8e1a-4d1b-9411-ed55205cc56c-kube-api-access-gpb6j podName:55baaf18-8e1a-4d1b-9411-ed55205cc56c nodeName:}" failed. No retries permitted until 2026-07-15 20:48:21.518109832 +0000 UTC m=+34.667147867 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gpb6j" (UniqueName: "kubernetes.io/projected/55baaf18-8e1a-4d1b-9411-ed55205cc56c-kube-api-access-gpb6j") pod "network-check-target-ngpfr" (UID: "55baaf18-8e1a-4d1b-9411-ed55205cc56c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 20:48:05.751124 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:05.751086    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwbql"
Jul 15 20:48:05.751124 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:05.751128    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngpfr"
Jul 15 20:48:05.751353 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:05.751086    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzw8h"
Jul 15 20:48:05.751353 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:05.751228    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwbql" podUID="42a42d42-83b0-441b-b7f2-3770f7523317"
Jul 15 20:48:05.751353 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:05.751328    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzw8h" podUID="b9d9267e-0c23-42b2-bb48-55ed14050fff"
Jul 15 20:48:05.751484 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:05.751425    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngpfr" podUID="55baaf18-8e1a-4d1b-9411-ed55205cc56c"
Jul 15 20:48:07.751399 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:07.751346    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwbql"
Jul 15 20:48:07.751988 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:07.751462    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzw8h"
Jul 15 20:48:07.751988 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:07.751471    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwbql" podUID="42a42d42-83b0-441b-b7f2-3770f7523317"
Jul 15 20:48:07.751988 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:07.751595    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzw8h" podUID="b9d9267e-0c23-42b2-bb48-55ed14050fff"
Jul 15 20:48:07.751988 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:07.751662    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngpfr"
Jul 15 20:48:07.751988 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:07.751734    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngpfr" podUID="55baaf18-8e1a-4d1b-9411-ed55205cc56c"
Jul 15 20:48:09.751390 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:09.750695    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzw8h"
Jul 15 20:48:09.751390 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:09.750823    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzw8h" podUID="b9d9267e-0c23-42b2-bb48-55ed14050fff"
Jul 15 20:48:09.751390 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:09.751252    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngpfr"
Jul 15 20:48:09.751390 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:09.751342    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngpfr" podUID="55baaf18-8e1a-4d1b-9411-ed55205cc56c"
Jul 15 20:48:09.752461 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:09.752003    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwbql"
Jul 15 20:48:09.752461 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:09.752129    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwbql" podUID="42a42d42-83b0-441b-b7f2-3770f7523317"
Jul 15 20:48:09.959438 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:09.959186    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wvh9c" event={"ID":"d786587a-4e93-443c-bfaa-334ced76d272","Type":"ContainerStarted","Data":"6cb9b8d40ee39093f709069e736d0169002cc06f36d3cabd7d510eedc325330f"}
Jul 15 20:48:09.964252 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:09.963828    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qvvx2" event={"ID":"147d4538-e5f1-47af-b61c-f07de8e0b0df","Type":"ContainerStarted","Data":"fd980bec96129c0f728b2464962619ebbbc64ad754f73c5992bf37793839668b"}
Jul 15 20:48:09.965923 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:09.965876    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7wpzz" event={"ID":"1bbd7d75-7a89-403d-983b-6534ab13e786","Type":"ContainerStarted","Data":"58e8a54c9785d3831ff1c6325858d2710012d1d42a4d51b4840e1bdeb1daa8ba"}
Jul 15 20:48:09.967694 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:09.967666    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6j89c" event={"ID":"ad23d165-aeec-409f-99bd-12c27029221b","Type":"ContainerStarted","Data":"6db1925a77cd14acb6b5169a88cb8d56fccf79dbd70d5af0156a081797933355"}
Jul 15 20:48:09.969312 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:09.969251    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wjxpz" event={"ID":"2dbf2724-61f8-4572-a2c1-906032db929d","Type":"ContainerStarted","Data":"87f0fbe33a40e4802ce8b11b41e605710a9dc4b09f93ab51208aacf99e793e8f"}
Jul 15 20:48:09.970824 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:09.970802    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wcchl" event={"ID":"1bfac382-72fc-400b-ab8b-9e4f53ab1d50","Type":"ContainerStarted","Data":"f4f3794c4c57ebf4c4816d5d84963016ad321311101ad4547329a55119ca1fc0"}
Jul 15 20:48:09.972596 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:09.972481    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8twzl" event={"ID":"e9ca36a8-de2e-472f-8b76-e6c4433f3ce4","Type":"ContainerStarted","Data":"20a3770c7e9ff22d1390e3671295b5fe09ec33802c6563fffaca0e184ba8a558"}
Jul 15 20:48:09.979776 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:09.979722    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-234.ec2.internal" podStartSLOduration=20.979704024 podStartE2EDuration="20.979704024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 20:47:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:47:52.93970102 +0000 UTC m=+6.088739077" watchObservedRunningTime="2026-07-15 20:48:09.979704024 +0000 UTC m=+23.128742083"
Jul 15 20:48:09.980010 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:09.979978    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-wvh9c" podStartSLOduration=3.2446945 podStartE2EDuration="21.979966903s" totalImagesPullingTime="18.735272403s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 20:47:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:48:09.979258936 +0000 UTC m=+23.128296993" watchObservedRunningTime="2026-07-15 20:48:09.979966903 +0000 UTC m=+23.129004961"
Jul 15 20:48:09.996487 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:09.996443    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-qvvx2" podStartSLOduration=3.333356474 podStartE2EDuration="21.996429416s" totalImagesPullingTime="18.663072942s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 20:47:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:48:09.995622552 +0000 UTC m=+23.144660636" watchObservedRunningTime="2026-07-15 20:48:09.996429416 +0000 UTC m=+23.145467470"
Jul 15 20:48:10.011736 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:10.011695    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-wcchl" podStartSLOduration=3.353717993 podStartE2EDuration="22.011679764s" totalImagesPullingTime="18.657961771s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 20:47:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:48:10.011319466 +0000 UTC m=+23.160357523" watchObservedRunningTime="2026-07-15 20:48:10.011679764 +0000 UTC m=+23.160717821"
Jul 15 20:48:10.029188 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:10.029145    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6j89c" podStartSLOduration=7.591703434 podStartE2EDuration="22.029127066s" totalImagesPullingTime="14.437423632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:47:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:48:10.028968015 +0000 UTC m=+23.178006071" watchObservedRunningTime="2026-07-15 20:48:10.029127066 +0000 UTC m=+23.178165122"
Jul 15 20:48:10.084029 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:10.083968    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7wpzz" podStartSLOduration=3.335571442 podStartE2EDuration="22.083947181s" totalImagesPullingTime="18.748375739s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 20:47:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:48:10.048724763 +0000 UTC m=+23.197762833" watchObservedRunningTime="2026-07-15 20:48:10.083947181 +0000 UTC m=+23.232985253"
Jul 15 20:48:10.978022 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:10.977822    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-652lk_f9a159bb-39c4-4cca-92ff-1c74d739e311/ovn-acl-logging/0.log"
Jul 15 20:48:10.978799 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:10.978352    2566 generic.go:356] "Generic (PLEG): container finished" podID="f9a159bb-39c4-4cca-92ff-1c74d739e311" containerID="a5d09cc5546cbc149db35b261d6742f9487f07a7dc5f52841e230a9fb1bfd194" exitCode=1
Jul 15 20:48:10.978799 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:10.978426    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-652lk" event={"ID":"f9a159bb-39c4-4cca-92ff-1c74d739e311","Type":"ContainerStarted","Data":"5aa063bcb67f51d9d9697f02aed348aa1825b76135ce2e31ea6da47f9e2d8555"}
Jul 15 20:48:10.978799 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:10.978546    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-652lk" event={"ID":"f9a159bb-39c4-4cca-92ff-1c74d739e311","Type":"ContainerStarted","Data":"204d1850af306f37560d34381f4c2dca870f7f1ad735480eb095ac63c7355a7f"}
Jul 15 20:48:10.978799 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:10.978596    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-652lk" event={"ID":"f9a159bb-39c4-4cca-92ff-1c74d739e311","Type":"ContainerStarted","Data":"5091314aef83f5c113cb2ccfce712222d407ea255cc6bfb62d71795b929c8a6a"}
Jul 15 20:48:10.978799 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:10.978643    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-652lk" event={"ID":"f9a159bb-39c4-4cca-92ff-1c74d739e311","Type":"ContainerStarted","Data":"6ab8e615bd49034790b1e0ce8f219c34956fb77775bab9d36ad4eef73349ecc5"}
Jul 15 20:48:10.978799 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:10.978687    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-652lk" event={"ID":"f9a159bb-39c4-4cca-92ff-1c74d739e311","Type":"ContainerDied","Data":"a5d09cc5546cbc149db35b261d6742f9487f07a7dc5f52841e230a9fb1bfd194"}
Jul 15 20:48:10.978799 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:10.978732    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-652lk" event={"ID":"f9a159bb-39c4-4cca-92ff-1c74d739e311","Type":"ContainerStarted","Data":"7735efb92fc32101d6b310774f7e7bc048428daa127ebe8c705e7d55cfd25da4"}
Jul 15 20:48:10.979811 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:10.979790    2566 generic.go:356] "Generic (PLEG): container finished" podID="2dbf2724-61f8-4572-a2c1-906032db929d" containerID="87f0fbe33a40e4802ce8b11b41e605710a9dc4b09f93ab51208aacf99e793e8f" exitCode=0
Jul 15 20:48:10.979948 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:10.979822    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wjxpz" event={"ID":"2dbf2724-61f8-4572-a2c1-906032db929d","Type":"ContainerDied","Data":"87f0fbe33a40e4802ce8b11b41e605710a9dc4b09f93ab51208aacf99e793e8f"}
Jul 15 20:48:11.118778 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:11.118751    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6j89c"
Jul 15 20:48:11.119274 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:11.119252    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6j89c"
Jul 15 20:48:11.332447 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:11.332426    2566 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 15 20:48:11.645855 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:11.645716    2566 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-15T20:48:11.332454406Z","UUID":"f5d14bd0-8125-41fa-962c-1de26965405f","Handler":null,"Name":"","Endpoint":""}
Jul 15 20:48:11.647787 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:11.647766    2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 15 20:48:11.647934 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:11.647808    2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 15 20:48:11.751101 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:11.751062    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwbql"
Jul 15 20:48:11.751101 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:11.751081    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngpfr"
Jul 15 20:48:11.751101 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:11.751063    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzw8h"
Jul 15 20:48:11.751439 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:11.751211    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngpfr" podUID="55baaf18-8e1a-4d1b-9411-ed55205cc56c"
Jul 15 20:48:11.751439 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:11.751265    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzw8h" podUID="b9d9267e-0c23-42b2-bb48-55ed14050fff"
Jul 15 20:48:11.751439 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:11.751351    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwbql" podUID="42a42d42-83b0-441b-b7f2-3770f7523317"
Jul 15 20:48:11.984132 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:11.984097    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-n2vtn" event={"ID":"3166ef5d-e008-44a0-90e4-4f9c39f2332d","Type":"ContainerStarted","Data":"a0226a91ed0073795be7b2d488f94c3f4ecf1b3b48ce903245ed0d4f3dff28df"}
Jul 15 20:48:11.986025 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:11.985991    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8twzl" event={"ID":"e9ca36a8-de2e-472f-8b76-e6c4433f3ce4","Type":"ContainerStarted","Data":"edfa58b28517c2246c834109522679accdd53d5e71cbe066bad45b46ab0145fd"}
Jul 15 20:48:11.986182 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:11.986166    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6j89c"
Jul 15 20:48:11.986640 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:11.986621    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6j89c"
Jul 15 20:48:12.020788 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:12.020749    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-n2vtn" podStartSLOduration=5.369467444 podStartE2EDuration="24.020735481s" totalImagesPullingTime="18.651268037s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 20:47:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:48:12.004487542 +0000 UTC m=+25.153525602" watchObservedRunningTime="2026-07-15 20:48:12.020735481 +0000 UTC m=+25.169773538"
Jul 15 20:48:12.991857 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:12.991837    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-652lk_f9a159bb-39c4-4cca-92ff-1c74d739e311/ovn-acl-logging/0.log"
Jul 15 20:48:12.992517 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:12.992416    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-652lk" event={"ID":"f9a159bb-39c4-4cca-92ff-1c74d739e311","Type":"ContainerStarted","Data":"83ffc600a886d84eeb07506f3f49c18f33db40e9dd7294eac84dae953789f9df"}
Jul 15 20:48:13.751092 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:13.750918    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwbql"
Jul 15 20:48:13.751397 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:13.750919    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzw8h"
Jul 15 20:48:13.751397 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:13.751206    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwbql" podUID="42a42d42-83b0-441b-b7f2-3770f7523317"
Jul 15 20:48:13.751397 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:13.750958    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngpfr"
Jul 15 20:48:13.751397 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:13.751272    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzw8h" podUID="b9d9267e-0c23-42b2-bb48-55ed14050fff"
Jul 15 20:48:13.751397 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:13.751330    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngpfr" podUID="55baaf18-8e1a-4d1b-9411-ed55205cc56c"
Jul 15 20:48:13.996987 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:13.996931    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8twzl" event={"ID":"e9ca36a8-de2e-472f-8b76-e6c4433f3ce4","Type":"ContainerStarted","Data":"2df772beba64e5f08a07197ae614b230cb649af709f6656e2acf21b922dbeaf7"}
Jul 15 20:48:14.018821 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:14.018733    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8twzl" podStartSLOduration=4.422145516 podStartE2EDuration="26.018718013s" totalImagesPullingTime="21.596572497s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 20:47:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:48:14.018319244 +0000 UTC m=+27.167357292" watchObservedRunningTime="2026-07-15 20:48:14.018718013 +0000 UTC m=+27.167756070"
Jul 15 20:48:15.751094 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:15.750913    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwbql"
Jul 15 20:48:15.752071 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:15.750913    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzw8h"
Jul 15 20:48:15.752071 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:15.751168    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwbql" podUID="42a42d42-83b0-441b-b7f2-3770f7523317"
Jul 15 20:48:15.752071 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:15.750908    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngpfr"
Jul 15 20:48:15.752071 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:15.751226    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzw8h" podUID="b9d9267e-0c23-42b2-bb48-55ed14050fff"
Jul 15 20:48:15.752071 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:15.751297    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngpfr" podUID="55baaf18-8e1a-4d1b-9411-ed55205cc56c"
Jul 15 20:48:16.003943 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:16.003864    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-652lk_f9a159bb-39c4-4cca-92ff-1c74d739e311/ovn-acl-logging/0.log"
Jul 15 20:48:16.004253 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:16.004227    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-652lk" event={"ID":"f9a159bb-39c4-4cca-92ff-1c74d739e311","Type":"ContainerStarted","Data":"480aff10ac7b92f442b099a125390713a6b1ab2781f20218b9c3e228755c0493"}
Jul 15 20:48:16.004480 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:16.004462    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:48:16.004580 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:16.004515    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:48:16.004689 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:16.004674    2566 scope.go:117] "RemoveContainer" containerID="a5d09cc5546cbc149db35b261d6742f9487f07a7dc5f52841e230a9fb1bfd194"
Jul 15 20:48:16.006220 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:16.006194    2566 generic.go:356] "Generic (PLEG): container finished" podID="2dbf2724-61f8-4572-a2c1-906032db929d" containerID="4450c7d65bfd664758943276ae0d2362d0a6640f41f0d44db88368e73b32acf1" exitCode=0
Jul 15 20:48:16.006330 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:16.006232    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wjxpz" event={"ID":"2dbf2724-61f8-4572-a2c1-906032db929d","Type":"ContainerDied","Data":"4450c7d65bfd664758943276ae0d2362d0a6640f41f0d44db88368e73b32acf1"}
Jul 15 20:48:16.020837 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:16.020819    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:48:17.017525 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:17.016540    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-652lk_f9a159bb-39c4-4cca-92ff-1c74d739e311/ovn-acl-logging/0.log"
Jul 15 20:48:17.018831 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:17.018798    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-652lk" event={"ID":"f9a159bb-39c4-4cca-92ff-1c74d739e311","Type":"ContainerStarted","Data":"e641717626a0177c1d2a356bb5d5436d72c4331eafc34742ad20b86d7185016c"}
Jul 15 20:48:17.019614 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:17.019526    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:48:17.038645 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:17.038617    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:48:17.065206 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:17.065156    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-652lk" podStartSLOduration=10.015600297 podStartE2EDuration="29.065137912s" totalImagesPullingTime="19.049537615s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 20:47:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:48:17.064479913 +0000 UTC m=+30.213517972" watchObservedRunningTime="2026-07-15 20:48:17.065137912 +0000 UTC m=+30.214175971"
Jul 15 20:48:17.221601 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:17.221562    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xwbql"]
Jul 15 20:48:17.221755 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:17.221684    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwbql"
Jul 15 20:48:17.221810 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:17.221774    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwbql" podUID="42a42d42-83b0-441b-b7f2-3770f7523317"
Jul 15 20:48:17.224443 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:17.224403    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zzw8h"]
Jul 15 20:48:17.224602 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:17.224530    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzw8h"
Jul 15 20:48:17.224651 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:17.224617    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzw8h" podUID="b9d9267e-0c23-42b2-bb48-55ed14050fff"
Jul 15 20:48:17.244247 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:17.244168    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ngpfr"]
Jul 15 20:48:17.244426 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:17.244273    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngpfr"
Jul 15 20:48:17.244426 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:17.244356    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngpfr" podUID="55baaf18-8e1a-4d1b-9411-ed55205cc56c"
Jul 15 20:48:18.023109 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:18.023078    2566 generic.go:356] "Generic (PLEG): container finished" podID="2dbf2724-61f8-4572-a2c1-906032db929d" containerID="ab548f99d62e9987a739cb4cf560ccd8b15e38d8766338884ec7cf03cdb6eb16" exitCode=0
Jul 15 20:48:18.023592 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:18.023174    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wjxpz" event={"ID":"2dbf2724-61f8-4572-a2c1-906032db929d","Type":"ContainerDied","Data":"ab548f99d62e9987a739cb4cf560ccd8b15e38d8766338884ec7cf03cdb6eb16"}
Jul 15 20:48:18.750984 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:18.750950    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngpfr"
Jul 15 20:48:18.750984 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:18.750980    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwbql"
Jul 15 20:48:18.751319 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:18.751051    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzw8h"
Jul 15 20:48:18.751319 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:18.751055    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngpfr" podUID="55baaf18-8e1a-4d1b-9411-ed55205cc56c"
Jul 15 20:48:18.751319 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:18.751127    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwbql" podUID="42a42d42-83b0-441b-b7f2-3770f7523317"
Jul 15 20:48:18.751319 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:18.751177    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzw8h" podUID="b9d9267e-0c23-42b2-bb48-55ed14050fff"
Jul 15 20:48:20.029260 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:20.029228    2566 generic.go:356] "Generic (PLEG): container finished" podID="2dbf2724-61f8-4572-a2c1-906032db929d" containerID="6a753a088ad56adfcaeac0aa7ce13465b84ca57f1698a440765c2ad3af8b496d" exitCode=0
Jul 15 20:48:20.029997 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:20.029290    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wjxpz" event={"ID":"2dbf2724-61f8-4572-a2c1-906032db929d","Type":"ContainerDied","Data":"6a753a088ad56adfcaeac0aa7ce13465b84ca57f1698a440765c2ad3af8b496d"}
Jul 15 20:48:20.751429 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:20.751389    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngpfr"
Jul 15 20:48:20.751611 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:20.751494    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwbql"
Jul 15 20:48:20.751611 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:20.751505    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngpfr" podUID="55baaf18-8e1a-4d1b-9411-ed55205cc56c"
Jul 15 20:48:20.751611 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:20.751530    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzw8h"
Jul 15 20:48:20.751776 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:20.751634    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwbql" podUID="42a42d42-83b0-441b-b7f2-3770f7523317"
Jul 15 20:48:20.751776 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:20.751714    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzw8h" podUID="b9d9267e-0c23-42b2-bb48-55ed14050fff"
Jul 15 20:48:21.337620 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:21.337581    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42a42d42-83b0-441b-b7f2-3770f7523317-metrics-certs\") pod \"network-metrics-daemon-xwbql\" (UID: \"42a42d42-83b0-441b-b7f2-3770f7523317\") " pod="openshift-multus/network-metrics-daemon-xwbql"
Jul 15 20:48:21.338194 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:21.337761    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 20:48:21.338194 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:21.337772    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9d9267e-0c23-42b2-bb48-55ed14050fff-original-pull-secret\") pod \"global-pull-secret-syncer-zzw8h\" (UID: \"b9d9267e-0c23-42b2-bb48-55ed14050fff\") " pod="kube-system/global-pull-secret-syncer-zzw8h"
Jul 15 20:48:21.338194 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:21.337836    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42a42d42-83b0-441b-b7f2-3770f7523317-metrics-certs podName:42a42d42-83b0-441b-b7f2-3770f7523317 nodeName:}" failed. No retries permitted until 2026-07-15 20:48:53.337814888 +0000 UTC m=+66.486852924 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/42a42d42-83b0-441b-b7f2-3770f7523317-metrics-certs") pod "network-metrics-daemon-xwbql" (UID: "42a42d42-83b0-441b-b7f2-3770f7523317") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 20:48:21.338194 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:21.337924    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 15 20:48:21.338194 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:21.337986    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9d9267e-0c23-42b2-bb48-55ed14050fff-original-pull-secret podName:b9d9267e-0c23-42b2-bb48-55ed14050fff nodeName:}" failed. No retries permitted until 2026-07-15 20:48:53.33797035 +0000 UTC m=+66.487008389 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b9d9267e-0c23-42b2-bb48-55ed14050fff-original-pull-secret") pod "global-pull-secret-syncer-zzw8h" (UID: "b9d9267e-0c23-42b2-bb48-55ed14050fff") : object "kube-system"/"original-pull-secret" not registered
Jul 15 20:48:21.539905 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:21.539867    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpb6j\" (UniqueName: \"kubernetes.io/projected/55baaf18-8e1a-4d1b-9411-ed55205cc56c-kube-api-access-gpb6j\") pod \"network-check-target-ngpfr\" (UID: \"55baaf18-8e1a-4d1b-9411-ed55205cc56c\") " pod="openshift-network-diagnostics/network-check-target-ngpfr"
Jul 15 20:48:21.540108 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:21.540050    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 15 20:48:21.540108 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:21.540074    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 15 20:48:21.540108 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:21.540087    2566 projected.go:196] Error preparing data for projected volume kube-api-access-gpb6j for pod openshift-network-diagnostics/network-check-target-ngpfr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 20:48:21.540268 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:21.540148    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55baaf18-8e1a-4d1b-9411-ed55205cc56c-kube-api-access-gpb6j podName:55baaf18-8e1a-4d1b-9411-ed55205cc56c nodeName:}" failed. No retries permitted until 2026-07-15 20:48:53.540132341 +0000 UTC m=+66.689170397 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gpb6j" (UniqueName: "kubernetes.io/projected/55baaf18-8e1a-4d1b-9411-ed55205cc56c-kube-api-access-gpb6j") pod "network-check-target-ngpfr" (UID: "55baaf18-8e1a-4d1b-9411-ed55205cc56c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 20:48:22.750578 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:22.750542    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngpfr"
Jul 15 20:48:22.751099 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:22.750543    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwbql"
Jul 15 20:48:22.751099 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:22.750657    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngpfr" podUID="55baaf18-8e1a-4d1b-9411-ed55205cc56c"
Jul 15 20:48:22.751099 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:22.750755    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwbql" podUID="42a42d42-83b0-441b-b7f2-3770f7523317"
Jul 15 20:48:22.751099 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:22.750544    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzw8h"
Jul 15 20:48:22.751099 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:22.750912    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzw8h" podUID="b9d9267e-0c23-42b2-bb48-55ed14050fff"
Jul 15 20:48:23.387934 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:23.387855    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:23.483557 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:23.483526    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:23.704504 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:23.704463    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-234.ec2.internal" event="NodeReady"
Jul 15 20:48:23.704675 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:23.704616    2566 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 15 20:48:23.756476 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:23.756445    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lmsms"]
Jul 15 20:48:23.759084 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:23.759059    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-t2nhn"]
Jul 15 20:48:23.759224 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:23.759193    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lmsms"
Jul 15 20:48:23.760779 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:23.760754    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-t2nhn"
Jul 15 20:48:23.762391 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:23.762364    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kll9v\""
Jul 15 20:48:23.762391 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:23.762383    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 15 20:48:23.762548 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:23.762365    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 15 20:48:23.763984 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:23.763965    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 15 20:48:23.764099 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:23.763988    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 15 20:48:23.764099 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:23.763967    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-pz297\""
Jul 15 20:48:23.764099 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:23.764001    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 15 20:48:23.772877 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:23.772854    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-t2nhn"]
Jul 15 20:48:23.775490 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:23.775468    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lmsms"]
Jul 15 20:48:23.857828 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:23.857793    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f53f2f8-84d0-4ee8-867c-f4735128a587-metrics-tls\") pod \"dns-default-lmsms\" (UID: \"7f53f2f8-84d0-4ee8-867c-f4735128a587\") " pod="openshift-dns/dns-default-lmsms"
Jul 15 20:48:23.857828 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:23.857827    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f53f2f8-84d0-4ee8-867c-f4735128a587-tmp-dir\") pod \"dns-default-lmsms\" (UID: \"7f53f2f8-84d0-4ee8-867c-f4735128a587\") " pod="openshift-dns/dns-default-lmsms"
Jul 15 20:48:23.858071 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:23.857848    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slqsf\" (UniqueName: \"kubernetes.io/projected/7f53f2f8-84d0-4ee8-867c-f4735128a587-kube-api-access-slqsf\") pod \"dns-default-lmsms\" (UID: \"7f53f2f8-84d0-4ee8-867c-f4735128a587\") " pod="openshift-dns/dns-default-lmsms"
Jul 15 20:48:23.858071 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:23.857921    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7f53f2f8-84d0-4ee8-867c-f4735128a587-config-volume\") pod \"dns-default-lmsms\" (UID: \"7f53f2f8-84d0-4ee8-867c-f4735128a587\") " pod="openshift-dns/dns-default-lmsms"
Jul 15 20:48:23.858071 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:23.857962    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ecc83405-3c34-4022-be1c-be802104e8b6-cert\") pod \"ingress-canary-t2nhn\" (UID: \"ecc83405-3c34-4022-be1c-be802104e8b6\") " pod="openshift-ingress-canary/ingress-canary-t2nhn"
Jul 15 20:48:23.858071 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:23.858031    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jld4n\" (UniqueName: \"kubernetes.io/projected/ecc83405-3c34-4022-be1c-be802104e8b6-kube-api-access-jld4n\") pod \"ingress-canary-t2nhn\" (UID: \"ecc83405-3c34-4022-be1c-be802104e8b6\") " pod="openshift-ingress-canary/ingress-canary-t2nhn"
Jul 15 20:48:23.958575 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:23.958400    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7f53f2f8-84d0-4ee8-867c-f4735128a587-config-volume\") pod \"dns-default-lmsms\" (UID: \"7f53f2f8-84d0-4ee8-867c-f4735128a587\") " pod="openshift-dns/dns-default-lmsms"
Jul 15 20:48:23.958681 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:23.958598    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ecc83405-3c34-4022-be1c-be802104e8b6-cert\") pod \"ingress-canary-t2nhn\" (UID: \"ecc83405-3c34-4022-be1c-be802104e8b6\") " pod="openshift-ingress-canary/ingress-canary-t2nhn"
Jul 15 20:48:23.958681 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:23.958658    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jld4n\" (UniqueName: \"kubernetes.io/projected/ecc83405-3c34-4022-be1c-be802104e8b6-kube-api-access-jld4n\") pod \"ingress-canary-t2nhn\" (UID: \"ecc83405-3c34-4022-be1c-be802104e8b6\") " pod="openshift-ingress-canary/ingress-canary-t2nhn"
Jul 15 20:48:23.958808 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:23.958786    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f53f2f8-84d0-4ee8-867c-f4735128a587-metrics-tls\") pod \"dns-default-lmsms\" (UID: \"7f53f2f8-84d0-4ee8-867c-f4735128a587\") " pod="openshift-dns/dns-default-lmsms"
Jul 15 20:48:23.958861 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:23.958829    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f53f2f8-84d0-4ee8-867c-f4735128a587-tmp-dir\") pod \"dns-default-lmsms\" (UID: \"7f53f2f8-84d0-4ee8-867c-f4735128a587\") " pod="openshift-dns/dns-default-lmsms"
Jul 15 20:48:23.958861 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:23.958854    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slqsf\" (UniqueName: \"kubernetes.io/projected/7f53f2f8-84d0-4ee8-867c-f4735128a587-kube-api-access-slqsf\") pod \"dns-default-lmsms\" (UID: \"7f53f2f8-84d0-4ee8-867c-f4735128a587\") " pod="openshift-dns/dns-default-lmsms"
Jul 15 20:48:23.959092 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:23.959072    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 15 20:48:23.959187 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:23.959125    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7f53f2f8-84d0-4ee8-867c-f4735128a587-config-volume\") pod \"dns-default-lmsms\" (UID: \"7f53f2f8-84d0-4ee8-867c-f4735128a587\") " pod="openshift-dns/dns-default-lmsms"
Jul 15 20:48:23.959187 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:23.959145    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f53f2f8-84d0-4ee8-867c-f4735128a587-metrics-tls podName:7f53f2f8-84d0-4ee8-867c-f4735128a587 nodeName:}" failed. No retries permitted until 2026-07-15 20:48:24.459127215 +0000 UTC m=+37.608165256 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7f53f2f8-84d0-4ee8-867c-f4735128a587-metrics-tls") pod "dns-default-lmsms" (UID: "7f53f2f8-84d0-4ee8-867c-f4735128a587") : secret "dns-default-metrics-tls" not found
Jul 15 20:48:23.959333 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:23.959309    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f53f2f8-84d0-4ee8-867c-f4735128a587-tmp-dir\") pod \"dns-default-lmsms\" (UID: \"7f53f2f8-84d0-4ee8-867c-f4735128a587\") " pod="openshift-dns/dns-default-lmsms"
Jul 15 20:48:23.963180 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:23.963156    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ecc83405-3c34-4022-be1c-be802104e8b6-cert\") pod \"ingress-canary-t2nhn\" (UID: \"ecc83405-3c34-4022-be1c-be802104e8b6\") " pod="openshift-ingress-canary/ingress-canary-t2nhn"
Jul 15 20:48:23.967113 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:23.967066    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jld4n\" (UniqueName: \"kubernetes.io/projected/ecc83405-3c34-4022-be1c-be802104e8b6-kube-api-access-jld4n\") pod \"ingress-canary-t2nhn\" (UID: \"ecc83405-3c34-4022-be1c-be802104e8b6\") " pod="openshift-ingress-canary/ingress-canary-t2nhn"
Jul 15 20:48:23.967694 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:23.967675    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slqsf\" (UniqueName: \"kubernetes.io/projected/7f53f2f8-84d0-4ee8-867c-f4735128a587-kube-api-access-slqsf\") pod \"dns-default-lmsms\" (UID: \"7f53f2f8-84d0-4ee8-867c-f4735128a587\") " pod="openshift-dns/dns-default-lmsms"
Jul 15 20:48:24.041512 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:24.041469    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wjxpz" event={"ID":"2dbf2724-61f8-4572-a2c1-906032db929d","Type":"ContainerStarted","Data":"30aca169113c80e9e3747befce23748b0fb2c37fff01c389c0ddded9190b92ad"}
Jul 15 20:48:24.081403 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:24.081369    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-t2nhn"
Jul 15 20:48:24.206018 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:24.205986    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-t2nhn"]
Jul 15 20:48:24.209697 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:48:24.209635    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecc83405_3c34_4022_be1c_be802104e8b6.slice/crio-3d3ad92ac3e7722075770e7ddea7cc7880b45d20dcd324c9e3195c80fd41a9d6 WatchSource:0}: Error finding container 3d3ad92ac3e7722075770e7ddea7cc7880b45d20dcd324c9e3195c80fd41a9d6: Status 404 returned error can't find the container with id 3d3ad92ac3e7722075770e7ddea7cc7880b45d20dcd324c9e3195c80fd41a9d6
Jul 15 20:48:24.462732 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:24.462651    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f53f2f8-84d0-4ee8-867c-f4735128a587-metrics-tls\") pod \"dns-default-lmsms\" (UID: \"7f53f2f8-84d0-4ee8-867c-f4735128a587\") " pod="openshift-dns/dns-default-lmsms"
Jul 15 20:48:24.462855 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:24.462759    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 15 20:48:24.462855 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:24.462815    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f53f2f8-84d0-4ee8-867c-f4735128a587-metrics-tls podName:7f53f2f8-84d0-4ee8-867c-f4735128a587 nodeName:}" failed. No retries permitted until 2026-07-15 20:48:25.462801623 +0000 UTC m=+38.611839658 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7f53f2f8-84d0-4ee8-867c-f4735128a587-metrics-tls") pod "dns-default-lmsms" (UID: "7f53f2f8-84d0-4ee8-867c-f4735128a587") : secret "dns-default-metrics-tls" not found
Jul 15 20:48:24.751396 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:24.751311    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwbql"
Jul 15 20:48:24.751562 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:24.751311    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngpfr"
Jul 15 20:48:24.751562 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:24.751311    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzw8h"
Jul 15 20:48:24.754684 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:24.754665    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 15 20:48:24.756185 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:24.756088    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lxhhk\""
Jul 15 20:48:24.756185 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:24.756137    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5sgcj\""
Jul 15 20:48:24.756185 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:24.756181    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 15 20:48:24.756409 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:24.756191    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 15 20:48:24.756409 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:24.756145    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 15 20:48:24.773853 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:24.773822    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:24.830180 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:24.830150    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:25.047327 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:25.047241    2566 generic.go:356] "Generic (PLEG): container finished" podID="2dbf2724-61f8-4572-a2c1-906032db929d" containerID="30aca169113c80e9e3747befce23748b0fb2c37fff01c389c0ddded9190b92ad" exitCode=0
Jul 15 20:48:25.047481 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:25.047339    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wjxpz" event={"ID":"2dbf2724-61f8-4572-a2c1-906032db929d","Type":"ContainerDied","Data":"30aca169113c80e9e3747befce23748b0fb2c37fff01c389c0ddded9190b92ad"}
Jul 15 20:48:25.048931 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:25.048504    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-t2nhn" event={"ID":"ecc83405-3c34-4022-be1c-be802104e8b6","Type":"ContainerStarted","Data":"3d3ad92ac3e7722075770e7ddea7cc7880b45d20dcd324c9e3195c80fd41a9d6"}
Jul 15 20:48:25.470056 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:25.469777    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f53f2f8-84d0-4ee8-867c-f4735128a587-metrics-tls\") pod \"dns-default-lmsms\" (UID: \"7f53f2f8-84d0-4ee8-867c-f4735128a587\") " pod="openshift-dns/dns-default-lmsms"
Jul 15 20:48:25.470211 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:25.469925    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 15 20:48:25.470211 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:25.470173    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f53f2f8-84d0-4ee8-867c-f4735128a587-metrics-tls podName:7f53f2f8-84d0-4ee8-867c-f4735128a587 nodeName:}" failed. No retries permitted until 2026-07-15 20:48:27.470155503 +0000 UTC m=+40.619193539 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7f53f2f8-84d0-4ee8-867c-f4735128a587-metrics-tls") pod "dns-default-lmsms" (UID: "7f53f2f8-84d0-4ee8-867c-f4735128a587") : secret "dns-default-metrics-tls" not found
Jul 15 20:48:25.497470 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:25.497436    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:25.597701 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:25.597670    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:25.806526 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:25.806442    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:26.058094 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:26.058003    2566 generic.go:356] "Generic (PLEG): container finished" podID="2dbf2724-61f8-4572-a2c1-906032db929d" containerID="6e451dc675092fd5b03eddbed016f1a81333f82b89e9bdbc6e413afc89e8c407" exitCode=0
Jul 15 20:48:26.058094 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:26.058049    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wjxpz" event={"ID":"2dbf2724-61f8-4572-a2c1-906032db929d","Type":"ContainerDied","Data":"6e451dc675092fd5b03eddbed016f1a81333f82b89e9bdbc6e413afc89e8c407"}
Jul 15 20:48:26.119976 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:26.119928    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:26.184385 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:26.184354    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:27.063692 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:27.063594    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wjxpz" event={"ID":"2dbf2724-61f8-4572-a2c1-906032db929d","Type":"ContainerStarted","Data":"8ef131d711486c61948bd66297d2180158786e30054ef98be825ccc9b6b073af"}
Jul 15 20:48:27.065141 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:27.065115    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-t2nhn" event={"ID":"ecc83405-3c34-4022-be1c-be802104e8b6","Type":"ContainerStarted","Data":"1cc2d99f9c0bc0e73e879f7b94d4df848365527f0925c0997ac0600060df290e"}
Jul 15 20:48:27.088813 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:27.088760    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wjxpz" podStartSLOduration=11.241903481 podStartE2EDuration="40.088744617s" totalImagesPullingTime="28.846841136s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 20:47:47 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:48:27.086966983 +0000 UTC m=+40.236005040" watchObservedRunningTime="2026-07-15 20:48:27.088744617 +0000 UTC m=+40.237782674"
Jul 15 20:48:27.106991 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:27.106947    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-t2nhn" podStartSLOduration=1.940181501 podStartE2EDuration="4.106931133s" totalImagesPullingTime="2.166749632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:48:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:48:27.106181551 +0000 UTC m=+40.255219611" watchObservedRunningTime="2026-07-15 20:48:27.106931133 +0000 UTC m=+40.255969189"
Jul 15 20:48:27.187165 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:27.187124    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:27.248006 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:27.247979    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:27.345154 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:27.345076    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:27.484023 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:27.483991    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f53f2f8-84d0-4ee8-867c-f4735128a587-metrics-tls\") pod \"dns-default-lmsms\" (UID: \"7f53f2f8-84d0-4ee8-867c-f4735128a587\") " pod="openshift-dns/dns-default-lmsms"
Jul 15 20:48:27.484163 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:27.484144    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 15 20:48:27.484218 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:27.484203    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f53f2f8-84d0-4ee8-867c-f4735128a587-metrics-tls podName:7f53f2f8-84d0-4ee8-867c-f4735128a587 nodeName:}" failed. No retries permitted until 2026-07-15 20:48:31.484185215 +0000 UTC m=+44.633223251 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7f53f2f8-84d0-4ee8-867c-f4735128a587-metrics-tls") pod "dns-default-lmsms" (UID: "7f53f2f8-84d0-4ee8-867c-f4735128a587") : secret "dns-default-metrics-tls" not found
Jul 15 20:48:28.295579 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:28.295551    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:28.396502 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:28.396463    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:28.599061 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:28.598979    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:28.704436 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:28.704407    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:29.564313 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:29.564277    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:29.710214 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:29.710182    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:29.827117 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:29.827038    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:29.865566 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:29.865540    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kwmkr"]
Jul 15 20:48:29.909881 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:29.909847    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kwmkr"]
Jul 15 20:48:29.910053 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:29.909988    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kwmkr"
Jul 15 20:48:29.912972 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:29.912949    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 15 20:48:29.913184 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:29.913001    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 15 20:48:29.913274 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:29.913246    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 15 20:48:29.914377 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:29.914358    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wc9r4\""
Jul 15 20:48:29.914487 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:29.914379    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 15 20:48:29.920162 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:29.920144    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:29.953404 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:29.953381    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-s9gwp"]
Jul 15 20:48:29.970174 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:29.970152    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-s9gwp"]
Jul 15 20:48:29.970283 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:29.970257    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-s9gwp"
Jul 15 20:48:29.973376 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:29.973349    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 15 20:48:29.973483 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:29.973377    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-6dvmq\""
Jul 15 20:48:29.973483 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:29.973435    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 15 20:48:29.986835 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:29.986815    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:30.000553 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:30.000529    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/075a9740-f176-4bb6-9d0a-9db3ba48fc75-crio-socket\") pod \"insights-runtime-extractor-kwmkr\" (UID: \"075a9740-f176-4bb6-9d0a-9db3ba48fc75\") " pod="openshift-insights/insights-runtime-extractor-kwmkr"
Jul 15 20:48:30.000628 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:30.000561    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/075a9740-f176-4bb6-9d0a-9db3ba48fc75-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kwmkr\" (UID: \"075a9740-f176-4bb6-9d0a-9db3ba48fc75\") " pod="openshift-insights/insights-runtime-extractor-kwmkr"
Jul 15 20:48:30.000672 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:30.000632    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/075a9740-f176-4bb6-9d0a-9db3ba48fc75-data-volume\") pod \"insights-runtime-extractor-kwmkr\" (UID: \"075a9740-f176-4bb6-9d0a-9db3ba48fc75\") " pod="openshift-insights/insights-runtime-extractor-kwmkr"
Jul 15 20:48:30.000672 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:30.000658    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzgbw\" (UniqueName: \"kubernetes.io/projected/075a9740-f176-4bb6-9d0a-9db3ba48fc75-kube-api-access-qzgbw\") pod \"insights-runtime-extractor-kwmkr\" (UID: \"075a9740-f176-4bb6-9d0a-9db3ba48fc75\") " pod="openshift-insights/insights-runtime-extractor-kwmkr"
Jul 15 20:48:30.000736 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:30.000698    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/075a9740-f176-4bb6-9d0a-9db3ba48fc75-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kwmkr\" (UID: \"075a9740-f176-4bb6-9d0a-9db3ba48fc75\") " pod="openshift-insights/insights-runtime-extractor-kwmkr"
Jul 15 20:48:30.090189 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:30.090099    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:30.101137 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:30.101111    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzgbw\" (UniqueName: \"kubernetes.io/projected/075a9740-f176-4bb6-9d0a-9db3ba48fc75-kube-api-access-qzgbw\") pod \"insights-runtime-extractor-kwmkr\" (UID: \"075a9740-f176-4bb6-9d0a-9db3ba48fc75\") " pod="openshift-insights/insights-runtime-extractor-kwmkr"
Jul 15 20:48:30.101218 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:30.101143    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/075a9740-f176-4bb6-9d0a-9db3ba48fc75-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kwmkr\" (UID: \"075a9740-f176-4bb6-9d0a-9db3ba48fc75\") " pod="openshift-insights/insights-runtime-extractor-kwmkr"
Jul 15 20:48:30.101218 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:30.101192    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/075a9740-f176-4bb6-9d0a-9db3ba48fc75-crio-socket\") pod \"insights-runtime-extractor-kwmkr\" (UID: \"075a9740-f176-4bb6-9d0a-9db3ba48fc75\") " pod="openshift-insights/insights-runtime-extractor-kwmkr"
Jul 15 20:48:30.101218 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:30.101211    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/075a9740-f176-4bb6-9d0a-9db3ba48fc75-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kwmkr\" (UID: \"075a9740-f176-4bb6-9d0a-9db3ba48fc75\") " pod="openshift-insights/insights-runtime-extractor-kwmkr"
Jul 15 20:48:30.101343 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:30.101236    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv57p\" (UniqueName: \"kubernetes.io/projected/6e3d4282-65a0-4a65-90c1-1da3cf585ff4-kube-api-access-pv57p\") pod \"migrator-7f59b54f6c-s9gwp\" (UID: \"6e3d4282-65a0-4a65-90c1-1da3cf585ff4\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-s9gwp"
Jul 15 20:48:30.101343 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:30.101282    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 15 20:48:30.101425 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:30.101345    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/075a9740-f176-4bb6-9d0a-9db3ba48fc75-kube-rbac-proxy-cm podName:075a9740-f176-4bb6-9d0a-9db3ba48fc75 nodeName:}" failed. No retries permitted until 2026-07-15 20:48:30.601329834 +0000 UTC m=+43.750367914 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/075a9740-f176-4bb6-9d0a-9db3ba48fc75-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kwmkr" (UID: "075a9740-f176-4bb6-9d0a-9db3ba48fc75") : configmap "kube-rbac-proxy" not found
Jul 15 20:48:30.101425 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:30.101346    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 15 20:48:30.101425 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:30.101410    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/075a9740-f176-4bb6-9d0a-9db3ba48fc75-insights-runtime-extractor-tls podName:075a9740-f176-4bb6-9d0a-9db3ba48fc75 nodeName:}" failed. No retries permitted until 2026-07-15 20:48:30.601394377 +0000 UTC m=+43.750432420 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/075a9740-f176-4bb6-9d0a-9db3ba48fc75-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kwmkr" (UID: "075a9740-f176-4bb6-9d0a-9db3ba48fc75") : secret "insights-runtime-extractor-tls" not found
Jul 15 20:48:30.101578 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:30.101426    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/075a9740-f176-4bb6-9d0a-9db3ba48fc75-crio-socket\") pod \"insights-runtime-extractor-kwmkr\" (UID: \"075a9740-f176-4bb6-9d0a-9db3ba48fc75\") " pod="openshift-insights/insights-runtime-extractor-kwmkr"
Jul 15 20:48:30.101578 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:30.101486    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/075a9740-f176-4bb6-9d0a-9db3ba48fc75-data-volume\") pod \"insights-runtime-extractor-kwmkr\" (UID: \"075a9740-f176-4bb6-9d0a-9db3ba48fc75\") " pod="openshift-insights/insights-runtime-extractor-kwmkr"
Jul 15 20:48:30.101801 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:30.101782    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/075a9740-f176-4bb6-9d0a-9db3ba48fc75-data-volume\") pod \"insights-runtime-extractor-kwmkr\" (UID: \"075a9740-f176-4bb6-9d0a-9db3ba48fc75\") " pod="openshift-insights/insights-runtime-extractor-kwmkr"
Jul 15 20:48:30.110878 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:30.110855    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzgbw\" (UniqueName: \"kubernetes.io/projected/075a9740-f176-4bb6-9d0a-9db3ba48fc75-kube-api-access-qzgbw\") pod \"insights-runtime-extractor-kwmkr\" (UID: \"075a9740-f176-4bb6-9d0a-9db3ba48fc75\") " pod="openshift-insights/insights-runtime-extractor-kwmkr"
Jul 15 20:48:30.201751 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:30.201717    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv57p\" (UniqueName: \"kubernetes.io/projected/6e3d4282-65a0-4a65-90c1-1da3cf585ff4-kube-api-access-pv57p\") pod \"migrator-7f59b54f6c-s9gwp\" (UID: \"6e3d4282-65a0-4a65-90c1-1da3cf585ff4\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-s9gwp"
Jul 15 20:48:30.201751 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:30.201736    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:30.215073 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:30.215041    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv57p\" (UniqueName: \"kubernetes.io/projected/6e3d4282-65a0-4a65-90c1-1da3cf585ff4-kube-api-access-pv57p\") pod \"migrator-7f59b54f6c-s9gwp\" (UID: \"6e3d4282-65a0-4a65-90c1-1da3cf585ff4\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-s9gwp"
Jul 15 20:48:30.278554 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:30.278521    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-s9gwp"
Jul 15 20:48:30.420066 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:30.420036    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-s9gwp"]
Jul 15 20:48:30.422935 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:48:30.422887    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e3d4282_65a0_4a65_90c1_1da3cf585ff4.slice/crio-35eb26d4a1082105db37528d37a49b4c335aca514518c99d959ddf9b9f69ba03 WatchSource:0}: Error finding container 35eb26d4a1082105db37528d37a49b4c335aca514518c99d959ddf9b9f69ba03: Status 404 returned error can't find the container with id 35eb26d4a1082105db37528d37a49b4c335aca514518c99d959ddf9b9f69ba03
Jul 15 20:48:30.532792 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:30.532742    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:30.603810 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:30.603746    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/075a9740-f176-4bb6-9d0a-9db3ba48fc75-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kwmkr\" (UID: \"075a9740-f176-4bb6-9d0a-9db3ba48fc75\") " pod="openshift-insights/insights-runtime-extractor-kwmkr"
Jul 15 20:48:30.603810 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:30.603808    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/075a9740-f176-4bb6-9d0a-9db3ba48fc75-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kwmkr\" (UID: \"075a9740-f176-4bb6-9d0a-9db3ba48fc75\") " pod="openshift-insights/insights-runtime-extractor-kwmkr"
Jul 15 20:48:30.604159 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:30.603874    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 15 20:48:30.604159 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:30.603930    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 15 20:48:30.604159 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:30.603979    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/075a9740-f176-4bb6-9d0a-9db3ba48fc75-kube-rbac-proxy-cm podName:075a9740-f176-4bb6-9d0a-9db3ba48fc75 nodeName:}" failed. No retries permitted until 2026-07-15 20:48:31.603958183 +0000 UTC m=+44.752996288 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/075a9740-f176-4bb6-9d0a-9db3ba48fc75-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kwmkr" (UID: "075a9740-f176-4bb6-9d0a-9db3ba48fc75") : configmap "kube-rbac-proxy" not found
Jul 15 20:48:30.604159 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:30.603994    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/075a9740-f176-4bb6-9d0a-9db3ba48fc75-insights-runtime-extractor-tls podName:075a9740-f176-4bb6-9d0a-9db3ba48fc75 nodeName:}" failed. No retries permitted until 2026-07-15 20:48:31.603987829 +0000 UTC m=+44.753025864 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/075a9740-f176-4bb6-9d0a-9db3ba48fc75-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kwmkr" (UID: "075a9740-f176-4bb6-9d0a-9db3ba48fc75") : secret "insights-runtime-extractor-tls" not found
Jul 15 20:48:30.637200 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:30.637174    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:31.076114 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:31.076074    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-s9gwp" event={"ID":"6e3d4282-65a0-4a65-90c1-1da3cf585ff4","Type":"ContainerStarted","Data":"35eb26d4a1082105db37528d37a49b4c335aca514518c99d959ddf9b9f69ba03"}
Jul 15 20:48:31.509229 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:31.509194    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f53f2f8-84d0-4ee8-867c-f4735128a587-metrics-tls\") pod \"dns-default-lmsms\" (UID: \"7f53f2f8-84d0-4ee8-867c-f4735128a587\") " pod="openshift-dns/dns-default-lmsms"
Jul 15 20:48:31.509418 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:31.509356    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 15 20:48:31.509460 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:31.509421    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f53f2f8-84d0-4ee8-867c-f4735128a587-metrics-tls podName:7f53f2f8-84d0-4ee8-867c-f4735128a587 nodeName:}" failed. No retries permitted until 2026-07-15 20:48:39.509406081 +0000 UTC m=+52.658444116 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7f53f2f8-84d0-4ee8-867c-f4735128a587-metrics-tls") pod "dns-default-lmsms" (UID: "7f53f2f8-84d0-4ee8-867c-f4735128a587") : secret "dns-default-metrics-tls" not found
Jul 15 20:48:31.610382 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:31.610346    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/075a9740-f176-4bb6-9d0a-9db3ba48fc75-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kwmkr\" (UID: \"075a9740-f176-4bb6-9d0a-9db3ba48fc75\") " pod="openshift-insights/insights-runtime-extractor-kwmkr"
Jul 15 20:48:31.610676 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:31.610416    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/075a9740-f176-4bb6-9d0a-9db3ba48fc75-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kwmkr\" (UID: \"075a9740-f176-4bb6-9d0a-9db3ba48fc75\") " pod="openshift-insights/insights-runtime-extractor-kwmkr"
Jul 15 20:48:31.610676 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:31.610470    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 15 20:48:31.610676 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:31.610504    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 15 20:48:31.610676 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:31.610541    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/075a9740-f176-4bb6-9d0a-9db3ba48fc75-kube-rbac-proxy-cm podName:075a9740-f176-4bb6-9d0a-9db3ba48fc75 nodeName:}" failed. No retries permitted until 2026-07-15 20:48:33.610522114 +0000 UTC m=+46.759560150 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/075a9740-f176-4bb6-9d0a-9db3ba48fc75-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kwmkr" (UID: "075a9740-f176-4bb6-9d0a-9db3ba48fc75") : configmap "kube-rbac-proxy" not found
Jul 15 20:48:31.610676 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:31.610556    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/075a9740-f176-4bb6-9d0a-9db3ba48fc75-insights-runtime-extractor-tls podName:075a9740-f176-4bb6-9d0a-9db3ba48fc75 nodeName:}" failed. No retries permitted until 2026-07-15 20:48:33.610549981 +0000 UTC m=+46.759588016 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/075a9740-f176-4bb6-9d0a-9db3ba48fc75-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kwmkr" (UID: "075a9740-f176-4bb6-9d0a-9db3ba48fc75") : secret "insights-runtime-extractor-tls" not found
Jul 15 20:48:31.705319 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:31.705287    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:31.833942 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:31.833835    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:32.041909 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:32.041854    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qvvx2_147d4538-e5f1-47af-b61c-f07de8e0b0df/dns-node-resolver/0.log"
Jul 15 20:48:32.119830 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:32.119759    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-4g86r"]
Jul 15 20:48:32.132667 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:32.132638    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-4g86r"
Jul 15 20:48:32.134456 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:32.134431    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-4g86r"]
Jul 15 20:48:32.135582 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:32.135559    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 15 20:48:32.137126 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:32.136965    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 15 20:48:32.137126 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:32.136965    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 15 20:48:32.137126 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:32.136967    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-xvn24\""
Jul 15 20:48:32.137126 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:32.137020    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 15 20:48:32.168096 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:32.168069    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:32.216964 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:32.216931    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6clc\" (UniqueName: \"kubernetes.io/projected/ce6c8970-4b7e-45d4-9aff-c0190bb806b0-kube-api-access-x6clc\") pod \"service-ca-79f68b9f4f-4g86r\" (UID: \"ce6c8970-4b7e-45d4-9aff-c0190bb806b0\") " pod="openshift-service-ca/service-ca-79f68b9f4f-4g86r"
Jul 15 20:48:32.217093 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:32.216973    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ce6c8970-4b7e-45d4-9aff-c0190bb806b0-signing-cabundle\") pod \"service-ca-79f68b9f4f-4g86r\" (UID: \"ce6c8970-4b7e-45d4-9aff-c0190bb806b0\") " pod="openshift-service-ca/service-ca-79f68b9f4f-4g86r"
Jul 15 20:48:32.217140 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:32.217102    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ce6c8970-4b7e-45d4-9aff-c0190bb806b0-signing-key\") pod \"service-ca-79f68b9f4f-4g86r\" (UID: \"ce6c8970-4b7e-45d4-9aff-c0190bb806b0\") " pod="openshift-service-ca/service-ca-79f68b9f4f-4g86r"
Jul 15 20:48:32.259609 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:32.259580    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:32.317601 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:32.317567    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ce6c8970-4b7e-45d4-9aff-c0190bb806b0-signing-key\") pod \"service-ca-79f68b9f4f-4g86r\" (UID: \"ce6c8970-4b7e-45d4-9aff-c0190bb806b0\") " pod="openshift-service-ca/service-ca-79f68b9f4f-4g86r"
Jul 15 20:48:32.317740 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:32.317630    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6clc\" (UniqueName: \"kubernetes.io/projected/ce6c8970-4b7e-45d4-9aff-c0190bb806b0-kube-api-access-x6clc\") pod \"service-ca-79f68b9f4f-4g86r\" (UID: \"ce6c8970-4b7e-45d4-9aff-c0190bb806b0\") " pod="openshift-service-ca/service-ca-79f68b9f4f-4g86r"
Jul 15 20:48:32.317740 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:32.317651    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ce6c8970-4b7e-45d4-9aff-c0190bb806b0-signing-cabundle\") pod \"service-ca-79f68b9f4f-4g86r\" (UID: \"ce6c8970-4b7e-45d4-9aff-c0190bb806b0\") " pod="openshift-service-ca/service-ca-79f68b9f4f-4g86r"
Jul 15 20:48:32.318232 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:32.318210    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ce6c8970-4b7e-45d4-9aff-c0190bb806b0-signing-cabundle\") pod \"service-ca-79f68b9f4f-4g86r\" (UID: \"ce6c8970-4b7e-45d4-9aff-c0190bb806b0\") " pod="openshift-service-ca/service-ca-79f68b9f4f-4g86r"
Jul 15 20:48:32.320003 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:32.319982    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ce6c8970-4b7e-45d4-9aff-c0190bb806b0-signing-key\") pod \"service-ca-79f68b9f4f-4g86r\" (UID: \"ce6c8970-4b7e-45d4-9aff-c0190bb806b0\") " pod="openshift-service-ca/service-ca-79f68b9f4f-4g86r"
Jul 15 20:48:32.327326 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:32.327305    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6clc\" (UniqueName: \"kubernetes.io/projected/ce6c8970-4b7e-45d4-9aff-c0190bb806b0-kube-api-access-x6clc\") pod \"service-ca-79f68b9f4f-4g86r\" (UID: \"ce6c8970-4b7e-45d4-9aff-c0190bb806b0\") " pod="openshift-service-ca/service-ca-79f68b9f4f-4g86r"
Jul 15 20:48:32.442599 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:32.442568    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-4g86r"
Jul 15 20:48:32.607764 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:32.607736    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-4g86r"]
Jul 15 20:48:32.610988 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:48:32.610946    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce6c8970_4b7e_45d4_9aff_c0190bb806b0.slice/crio-5082f97478e445c50c1552ff8135047dc9164f08fdac79b15d52e8402b4273c8 WatchSource:0}: Error finding container 5082f97478e445c50c1552ff8135047dc9164f08fdac79b15d52e8402b4273c8: Status 404 returned error can't find the container with id 5082f97478e445c50c1552ff8135047dc9164f08fdac79b15d52e8402b4273c8
Jul 15 20:48:33.042257 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:33.042230    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wcchl_1bfac382-72fc-400b-ab8b-9e4f53ab1d50/node-ca/0.log"
Jul 15 20:48:33.081837 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:33.081796    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-4g86r" event={"ID":"ce6c8970-4b7e-45d4-9aff-c0190bb806b0","Type":"ContainerStarted","Data":"5082f97478e445c50c1552ff8135047dc9164f08fdac79b15d52e8402b4273c8"}
Jul 15 20:48:33.083504 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:33.083486    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-s9gwp" event={"ID":"6e3d4282-65a0-4a65-90c1-1da3cf585ff4","Type":"ContainerStarted","Data":"4b17d5743edfb5a067bd266c8e68de71b38e7453b9e12f1ab7ddb6453ba2a5ad"}
Jul 15 20:48:33.083606 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:33.083523    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-s9gwp" event={"ID":"6e3d4282-65a0-4a65-90c1-1da3cf585ff4","Type":"ContainerStarted","Data":"d5979e41118ca10a08766188a07b39012095c441bad6201c33c9d8a77b1bfeba"}
Jul 15 20:48:33.106752 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:33.106708    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-s9gwp" podStartSLOduration=2.17335233 podStartE2EDuration="4.106693181s" totalImagesPullingTime="1.933340851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:48:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:48:33.106005164 +0000 UTC m=+46.255043220" watchObservedRunningTime="2026-07-15 20:48:33.106693181 +0000 UTC m=+46.255731577"
Jul 15 20:48:33.250724 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:33.250692    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:33.627524 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:33.627483    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/075a9740-f176-4bb6-9d0a-9db3ba48fc75-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kwmkr\" (UID: \"075a9740-f176-4bb6-9d0a-9db3ba48fc75\") " pod="openshift-insights/insights-runtime-extractor-kwmkr"
Jul 15 20:48:33.628007 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:33.627570    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/075a9740-f176-4bb6-9d0a-9db3ba48fc75-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kwmkr\" (UID: \"075a9740-f176-4bb6-9d0a-9db3ba48fc75\") " pod="openshift-insights/insights-runtime-extractor-kwmkr"
Jul 15 20:48:33.628007 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:33.627620    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 15 20:48:33.628007 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:33.627702    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/075a9740-f176-4bb6-9d0a-9db3ba48fc75-kube-rbac-proxy-cm podName:075a9740-f176-4bb6-9d0a-9db3ba48fc75 nodeName:}" failed. No retries permitted until 2026-07-15 20:48:37.627681955 +0000 UTC m=+50.776719995 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/075a9740-f176-4bb6-9d0a-9db3ba48fc75-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kwmkr" (UID: "075a9740-f176-4bb6-9d0a-9db3ba48fc75") : configmap "kube-rbac-proxy" not found
Jul 15 20:48:33.628007 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:33.627712    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 15 20:48:33.628007 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:33.627749    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/075a9740-f176-4bb6-9d0a-9db3ba48fc75-insights-runtime-extractor-tls podName:075a9740-f176-4bb6-9d0a-9db3ba48fc75 nodeName:}" failed. No retries permitted until 2026-07-15 20:48:37.627737664 +0000 UTC m=+50.776775710 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/075a9740-f176-4bb6-9d0a-9db3ba48fc75-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kwmkr" (UID: "075a9740-f176-4bb6-9d0a-9db3ba48fc75") : secret "insights-runtime-extractor-tls" not found
Jul 15 20:48:33.643861 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:33.643833    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-t2nhn_ecc83405-3c34-4022-be1c-be802104e8b6/serve-healthcheck-canary/0.log"
Jul 15 20:48:35.914742 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:35.914708    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:36.092266 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:36.092231    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-4g86r" event={"ID":"ce6c8970-4b7e-45d4-9aff-c0190bb806b0","Type":"ContainerStarted","Data":"7c4652eb5983ad84facebf75e3cfcf9d0afe536b455b5fc14f30fa5685c24b04"}
Jul 15 20:48:36.113558 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:36.113507    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-4g86r" podStartSLOduration=1.571842174 podStartE2EDuration="4.113491855s" totalImagesPullingTime="2.541649681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:48:36.113189725 +0000 UTC m=+49.262227783" watchObservedRunningTime="2026-07-15 20:48:36.113491855 +0000 UTC m=+49.262530267"
Jul 15 20:48:36.273625 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:36.273588    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:36.431338 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:36.431297    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:36.535926 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:36.535821    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:36.649170 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:36.649136    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:36.759975 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:36.759942    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:36.847403 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:36.847317    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:36.970251 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:36.970211    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:37.660750 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:37.660716    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/075a9740-f176-4bb6-9d0a-9db3ba48fc75-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kwmkr\" (UID: \"075a9740-f176-4bb6-9d0a-9db3ba48fc75\") " pod="openshift-insights/insights-runtime-extractor-kwmkr"
Jul 15 20:48:37.660942 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:37.660822    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/075a9740-f176-4bb6-9d0a-9db3ba48fc75-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kwmkr\" (UID: \"075a9740-f176-4bb6-9d0a-9db3ba48fc75\") " pod="openshift-insights/insights-runtime-extractor-kwmkr"
Jul 15 20:48:37.660994 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:37.660939    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 15 20:48:37.661028 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:48:37.661015    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/075a9740-f176-4bb6-9d0a-9db3ba48fc75-insights-runtime-extractor-tls podName:075a9740-f176-4bb6-9d0a-9db3ba48fc75 nodeName:}" failed. No retries permitted until 2026-07-15 20:48:45.660994739 +0000 UTC m=+58.810032792 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/075a9740-f176-4bb6-9d0a-9db3ba48fc75-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kwmkr" (UID: "075a9740-f176-4bb6-9d0a-9db3ba48fc75") : secret "insights-runtime-extractor-tls" not found
Jul 15 20:48:37.661366 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:37.661346    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/075a9740-f176-4bb6-9d0a-9db3ba48fc75-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kwmkr\" (UID: \"075a9740-f176-4bb6-9d0a-9db3ba48fc75\") " pod="openshift-insights/insights-runtime-extractor-kwmkr"
Jul 15 20:48:37.790352 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:37.790314    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:38.196082 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:38.196048    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:38.624251 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:38.624176    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:38.779271 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:38.779234    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:39.196681 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:39.196647    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:39.575848 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:39.575758    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f53f2f8-84d0-4ee8-867c-f4735128a587-metrics-tls\") pod \"dns-default-lmsms\" (UID: \"7f53f2f8-84d0-4ee8-867c-f4735128a587\") " pod="openshift-dns/dns-default-lmsms"
Jul 15 20:48:39.578074 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:39.578047    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f53f2f8-84d0-4ee8-867c-f4735128a587-metrics-tls\") pod \"dns-default-lmsms\" (UID: \"7f53f2f8-84d0-4ee8-867c-f4735128a587\") " pod="openshift-dns/dns-default-lmsms"
Jul 15 20:48:39.674508 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:39.674472    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lmsms"
Jul 15 20:48:39.760809 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:39.760752    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:39.820524 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:39.820491    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lmsms"]
Jul 15 20:48:39.824492 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:48:39.824458    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f53f2f8_84d0_4ee8_867c_f4735128a587.slice/crio-9d1134992885f2f2dc6f50f1366c1210f0aec4e52a3d058761d6750ed180f1b4 WatchSource:0}: Error finding container 9d1134992885f2f2dc6f50f1366c1210f0aec4e52a3d058761d6750ed180f1b4: Status 404 returned error can't find the container with id 9d1134992885f2f2dc6f50f1366c1210f0aec4e52a3d058761d6750ed180f1b4
Jul 15 20:48:39.962093 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:39.962060    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:40.103922 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:40.103818    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lmsms" event={"ID":"7f53f2f8-84d0-4ee8-867c-f4735128a587","Type":"ContainerStarted","Data":"9d1134992885f2f2dc6f50f1366c1210f0aec4e52a3d058761d6750ed180f1b4"}
Jul 15 20:48:40.153277 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:40.153248    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:40.253673 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:40.253641    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:40.369952 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:40.369859    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:40.830370 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:40.830315    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:40.967753 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:40.967719    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:41.382513 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:41.382473    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:41.776275 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:41.776235    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:42.161506 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:42.161474    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:42.374829 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:42.374729    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:42.452764 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:42.452719    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:42.834179 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:42.834144    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:42.971700 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:42.971660    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:43.117470 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:43.117380    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lmsms" event={"ID":"7f53f2f8-84d0-4ee8-867c-f4735128a587","Type":"ContainerStarted","Data":"c3e828368c6d632267b8652605d563e155ee870d1ad06e77654376af0f3a99a2"}
Jul 15 20:48:43.117470 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:43.117437    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lmsms" event={"ID":"7f53f2f8-84d0-4ee8-867c-f4735128a587","Type":"ContainerStarted","Data":"c77b64534de64e03d83640a5559d3d314e304b4b2453c00cf2e0ed01c52fa27b"}
Jul 15 20:48:43.117655 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:43.117478    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lmsms"
Jul 15 20:48:43.140257 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:43.140207    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-lmsms" podStartSLOduration=17.968908571 podStartE2EDuration="20.140191231s" totalImagesPullingTime="2.17128266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:48:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:48:43.138436878 +0000 UTC m=+56.287474948" watchObservedRunningTime="2026-07-15 20:48:43.140191231 +0000 UTC m=+56.289229288"
Jul 15 20:48:43.556313 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:43.556259    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:43.763621 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:43.763588    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:43.965792 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:43.965747    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:44.362246 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:44.362132    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:44.765044 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:44.764984    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:45.359208 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:45.359171    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:45.725958 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:45.725923    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/075a9740-f176-4bb6-9d0a-9db3ba48fc75-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kwmkr\" (UID: \"075a9740-f176-4bb6-9d0a-9db3ba48fc75\") " pod="openshift-insights/insights-runtime-extractor-kwmkr"
Jul 15 20:48:45.728208 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:45.728186    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/075a9740-f176-4bb6-9d0a-9db3ba48fc75-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kwmkr\" (UID: \"075a9740-f176-4bb6-9d0a-9db3ba48fc75\") " pod="openshift-insights/insights-runtime-extractor-kwmkr"
Jul 15 20:48:45.751932 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:45.751882    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:45.819376 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:45.819342    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kwmkr"
Jul 15 20:48:45.950493 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:45.950451    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kwmkr"]
Jul 15 20:48:45.954444 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:48:45.954413    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod075a9740_f176_4bb6_9d0a_9db3ba48fc75.slice/crio-e9be8323d537653ab73de51525f2e887340d562971f3fa6e2e4c7655c3343d8c WatchSource:0}: Error finding container e9be8323d537653ab73de51525f2e887340d562971f3fa6e2e4c7655c3343d8c: Status 404 returned error can't find the container with id e9be8323d537653ab73de51525f2e887340d562971f3fa6e2e4c7655c3343d8c
Jul 15 20:48:46.129171 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:46.129089    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kwmkr" event={"ID":"075a9740-f176-4bb6-9d0a-9db3ba48fc75","Type":"ContainerStarted","Data":"e9be8323d537653ab73de51525f2e887340d562971f3fa6e2e4c7655c3343d8c"}
Jul 15 20:48:46.151529 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:46.151495    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:46.566292 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:46.566261    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:46.719082 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:46.719013    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:46.813558 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:46.813474    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:47.003686 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:47.003618    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:49.041603 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:49.041577    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-652lk"
Jul 15 20:48:49.304790 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:49.304715    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:52.152954 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:52.152920    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kwmkr" event={"ID":"075a9740-f176-4bb6-9d0a-9db3ba48fc75","Type":"ContainerStarted","Data":"1293060295b17bff5a3ab7609d8caa73374c155d25b45098e506cb660ccaf58c"}
Jul 15 20:48:53.123502 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:53.123465    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lmsms"
Jul 15 20:48:53.387110 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:53.387019    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42a42d42-83b0-441b-b7f2-3770f7523317-metrics-certs\") pod \"network-metrics-daemon-xwbql\" (UID: \"42a42d42-83b0-441b-b7f2-3770f7523317\") " pod="openshift-multus/network-metrics-daemon-xwbql"
Jul 15 20:48:53.387532 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:53.387129    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9d9267e-0c23-42b2-bb48-55ed14050fff-original-pull-secret\") pod \"global-pull-secret-syncer-zzw8h\" (UID: \"b9d9267e-0c23-42b2-bb48-55ed14050fff\") " pod="kube-system/global-pull-secret-syncer-zzw8h"
Jul 15 20:48:53.390292 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:53.390266    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 15 20:48:53.390977 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:53.390918    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 15 20:48:53.400584 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:53.400558    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9d9267e-0c23-42b2-bb48-55ed14050fff-original-pull-secret\") pod \"global-pull-secret-syncer-zzw8h\" (UID: \"b9d9267e-0c23-42b2-bb48-55ed14050fff\") " pod="kube-system/global-pull-secret-syncer-zzw8h"
Jul 15 20:48:53.400584 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:53.400578    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42a42d42-83b0-441b-b7f2-3770f7523317-metrics-certs\") pod \"network-metrics-daemon-xwbql\" (UID: \"42a42d42-83b0-441b-b7f2-3770f7523317\") " pod="openshift-multus/network-metrics-daemon-xwbql"
Jul 15 20:48:53.566550 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:53.566519    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lxhhk\""
Jul 15 20:48:53.574338 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:53.574309    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwbql"
Jul 15 20:48:53.576601 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:53.576574    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzw8h"
Jul 15 20:48:53.589440 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:53.589411    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpb6j\" (UniqueName: \"kubernetes.io/projected/55baaf18-8e1a-4d1b-9411-ed55205cc56c-kube-api-access-gpb6j\") pod \"network-check-target-ngpfr\" (UID: \"55baaf18-8e1a-4d1b-9411-ed55205cc56c\") " pod="openshift-network-diagnostics/network-check-target-ngpfr"
Jul 15 20:48:53.592606 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:53.592585    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 15 20:48:53.603231 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:53.603195    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 15 20:48:53.613421 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:53.613398    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpb6j\" (UniqueName: \"kubernetes.io/projected/55baaf18-8e1a-4d1b-9411-ed55205cc56c-kube-api-access-gpb6j\") pod \"network-check-target-ngpfr\" (UID: \"55baaf18-8e1a-4d1b-9411-ed55205cc56c\") " pod="openshift-network-diagnostics/network-check-target-ngpfr"
Jul 15 20:48:53.872913 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:53.872873    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5sgcj\""
Jul 15 20:48:53.880409 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:53.880384    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngpfr"
Jul 15 20:48:54.547604 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:54.547574    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ngpfr"]
Jul 15 20:48:54.551117 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:48:54.551089    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55baaf18_8e1a_4d1b_9411_ed55205cc56c.slice/crio-5d1095df1d4797bc1915dcf32e902cb501ef46d77b530abb549b8d484984b355 WatchSource:0}: Error finding container 5d1095df1d4797bc1915dcf32e902cb501ef46d77b530abb549b8d484984b355: Status 404 returned error can't find the container with id 5d1095df1d4797bc1915dcf32e902cb501ef46d77b530abb549b8d484984b355
Jul 15 20:48:54.751214 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:54.751175    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zzw8h"]
Jul 15 20:48:54.754750 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:54.754725    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xwbql"]
Jul 15 20:48:54.755203 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:48:54.755174    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9d9267e_0c23_42b2_bb48_55ed14050fff.slice/crio-e14a191bdf5036aa9919b748f07525e3939a74cc42762606541f7fe196117970 WatchSource:0}: Error finding container e14a191bdf5036aa9919b748f07525e3939a74cc42762606541f7fe196117970: Status 404 returned error can't find the container with id e14a191bdf5036aa9919b748f07525e3939a74cc42762606541f7fe196117970
Jul 15 20:48:54.758093 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:48:54.758071    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42a42d42_83b0_441b_b7f2_3770f7523317.slice/crio-a6c9011d56d452f9064519f041a71fec25bb0d69d9a7fe3379411c1869ea36bc WatchSource:0}: Error finding container a6c9011d56d452f9064519f041a71fec25bb0d69d9a7fe3379411c1869ea36bc: Status 404 returned error can't find the container with id a6c9011d56d452f9064519f041a71fec25bb0d69d9a7fe3379411c1869ea36bc
Jul 15 20:48:55.165778 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:55.165652    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kwmkr" event={"ID":"075a9740-f176-4bb6-9d0a-9db3ba48fc75","Type":"ContainerStarted","Data":"39976822a02b08def1b6adf7cd51f2474425cabc1f12fd0d6b12d6b8c95af9e6"}
Jul 15 20:48:55.167264 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:55.167202    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xwbql" event={"ID":"42a42d42-83b0-441b-b7f2-3770f7523317","Type":"ContainerStarted","Data":"a6c9011d56d452f9064519f041a71fec25bb0d69d9a7fe3379411c1869ea36bc"}
Jul 15 20:48:55.168746 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:55.168718    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ngpfr" event={"ID":"55baaf18-8e1a-4d1b-9411-ed55205cc56c","Type":"ContainerStarted","Data":"5d1095df1d4797bc1915dcf32e902cb501ef46d77b530abb549b8d484984b355"}
Jul 15 20:48:55.169871 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:55.169823    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zzw8h" event={"ID":"b9d9267e-0c23-42b2-bb48-55ed14050fff","Type":"ContainerStarted","Data":"e14a191bdf5036aa9919b748f07525e3939a74cc42762606541f7fe196117970"}
Jul 15 20:48:57.181393 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:57.181308    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xwbql" event={"ID":"42a42d42-83b0-441b-b7f2-3770f7523317","Type":"ContainerStarted","Data":"726d80e3f7229659014db1606017c34a306c01ade391d22132df7c091ad2963b"}
Jul 15 20:48:57.181393 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:57.181382    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xwbql" event={"ID":"42a42d42-83b0-441b-b7f2-3770f7523317","Type":"ContainerStarted","Data":"b65e01b94b130244991aa52b5a730e82948b20a0d0aa376f332bfffc1c8bf505"}
Jul 15 20:48:57.200592 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:57.200526    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xwbql" podStartSLOduration=68.445745892 podStartE2EDuration="1m10.200505825s" totalImagesPullingTime="1.754759933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:47:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:48:57.199403515 +0000 UTC m=+70.348441573" watchObservedRunningTime="2026-07-15 20:48:57.200505825 +0000 UTC m=+70.349543882"
Jul 15 20:48:58.681446 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:58.681404    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86dd496bfc-kzhj5"]
Jul 15 20:48:58.691125 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:58.691097    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5b688c88-krjrp"]
Jul 15 20:48:58.691434 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:58.691407    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86dd496bfc-kzhj5"
Jul 15 20:48:58.692991 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:58.692970    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5b688c88-krjrp"
Jul 15 20:48:58.694346 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:58.694328    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 15 20:48:58.694593 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:58.694574    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 15 20:48:58.695176 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:58.695152    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 15 20:48:58.696595 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:58.696574    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 15 20:48:58.696723 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:58.696574    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 15 20:48:58.696723 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:58.696710    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 15 20:48:58.696875 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:58.696859    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 15 20:48:58.697868 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:58.697847    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 15 20:48:58.705566 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:58.705545    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5b688c88-krjrp"]
Jul 15 20:48:58.727882 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:58.727850    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b8ff996f-3019-49a3-a67a-89beb08d8676-klusterlet-config\") pod \"klusterlet-addon-workmgr-7f5b688c88-krjrp\" (UID: \"b8ff996f-3019-49a3-a67a-89beb08d8676\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5b688c88-krjrp"
Jul 15 20:48:58.728047 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:58.727936    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8ff996f-3019-49a3-a67a-89beb08d8676-tmp\") pod \"klusterlet-addon-workmgr-7f5b688c88-krjrp\" (UID: \"b8ff996f-3019-49a3-a67a-89beb08d8676\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5b688c88-krjrp"
Jul 15 20:48:58.728047 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:58.727967    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2b1a8d32-166f-4e11-8089-856faaac212d-ca\") pod \"cluster-proxy-proxy-agent-86dd496bfc-kzhj5\" (UID: \"2b1a8d32-166f-4e11-8089-856faaac212d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86dd496bfc-kzhj5"
Jul 15 20:48:58.728047 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:58.727998    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb2jb\" (UniqueName: \"kubernetes.io/projected/2b1a8d32-166f-4e11-8089-856faaac212d-kube-api-access-xb2jb\") pod \"cluster-proxy-proxy-agent-86dd496bfc-kzhj5\" (UID: \"2b1a8d32-166f-4e11-8089-856faaac212d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86dd496bfc-kzhj5"
Jul 15 20:48:58.728199 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:58.728045    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hppn\" (UniqueName: \"kubernetes.io/projected/b8ff996f-3019-49a3-a67a-89beb08d8676-kube-api-access-5hppn\") pod \"klusterlet-addon-workmgr-7f5b688c88-krjrp\" (UID: \"b8ff996f-3019-49a3-a67a-89beb08d8676\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5b688c88-krjrp"
Jul 15 20:48:58.728199 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:58.728081    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2b1a8d32-166f-4e11-8089-856faaac212d-hub\") pod \"cluster-proxy-proxy-agent-86dd496bfc-kzhj5\" (UID: \"2b1a8d32-166f-4e11-8089-856faaac212d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86dd496bfc-kzhj5"
Jul 15 20:48:58.728199 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:58.728104    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2b1a8d32-166f-4e11-8089-856faaac212d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-86dd496bfc-kzhj5\" (UID: \"2b1a8d32-166f-4e11-8089-856faaac212d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86dd496bfc-kzhj5"
Jul 15 20:48:58.728199 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:58.728182    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b1a8d32-166f-4e11-8089-856faaac212d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-86dd496bfc-kzhj5\" (UID: \"2b1a8d32-166f-4e11-8089-856faaac212d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86dd496bfc-kzhj5"
Jul 15 20:48:58.728363 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:58.728249    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2b1a8d32-166f-4e11-8089-856faaac212d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-86dd496bfc-kzhj5\" (UID: \"2b1a8d32-166f-4e11-8089-856faaac212d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86dd496bfc-kzhj5"
Jul 15 20:48:58.775409 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:58.775372    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86dd496bfc-kzhj5"]
Jul 15 20:48:58.819354 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:58.819321    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:58.829360 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:58.829330    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2b1a8d32-166f-4e11-8089-856faaac212d-hub\") pod \"cluster-proxy-proxy-agent-86dd496bfc-kzhj5\" (UID: \"2b1a8d32-166f-4e11-8089-856faaac212d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86dd496bfc-kzhj5"
Jul 15 20:48:58.829519 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:58.829371    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2b1a8d32-166f-4e11-8089-856faaac212d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-86dd496bfc-kzhj5\" (UID: \"2b1a8d32-166f-4e11-8089-856faaac212d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86dd496bfc-kzhj5"
Jul 15 20:48:58.829519 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:58.829403    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b1a8d32-166f-4e11-8089-856faaac212d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-86dd496bfc-kzhj5\" (UID: \"2b1a8d32-166f-4e11-8089-856faaac212d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86dd496bfc-kzhj5"
Jul 15 20:48:58.829519 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:58.829456    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2b1a8d32-166f-4e11-8089-856faaac212d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-86dd496bfc-kzhj5\" (UID: \"2b1a8d32-166f-4e11-8089-856faaac212d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86dd496bfc-kzhj5"
Jul 15 20:48:58.829519 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:58.829494    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b8ff996f-3019-49a3-a67a-89beb08d8676-klusterlet-config\") pod \"klusterlet-addon-workmgr-7f5b688c88-krjrp\" (UID: \"b8ff996f-3019-49a3-a67a-89beb08d8676\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5b688c88-krjrp"
Jul 15 20:48:58.829722 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:58.829521    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8ff996f-3019-49a3-a67a-89beb08d8676-tmp\") pod \"klusterlet-addon-workmgr-7f5b688c88-krjrp\" (UID: \"b8ff996f-3019-49a3-a67a-89beb08d8676\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5b688c88-krjrp"
Jul 15 20:48:58.829722 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:58.829543    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2b1a8d32-166f-4e11-8089-856faaac212d-ca\") pod \"cluster-proxy-proxy-agent-86dd496bfc-kzhj5\" (UID: \"2b1a8d32-166f-4e11-8089-856faaac212d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86dd496bfc-kzhj5"
Jul 15 20:48:58.829722 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:58.829569    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb2jb\" (UniqueName: \"kubernetes.io/projected/2b1a8d32-166f-4e11-8089-856faaac212d-kube-api-access-xb2jb\") pod \"cluster-proxy-proxy-agent-86dd496bfc-kzhj5\" (UID: \"2b1a8d32-166f-4e11-8089-856faaac212d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86dd496bfc-kzhj5"
Jul 15 20:48:58.829722 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:58.829596    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hppn\" (UniqueName: \"kubernetes.io/projected/b8ff996f-3019-49a3-a67a-89beb08d8676-kube-api-access-5hppn\") pod \"klusterlet-addon-workmgr-7f5b688c88-krjrp\" (UID: \"b8ff996f-3019-49a3-a67a-89beb08d8676\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5b688c88-krjrp"
Jul 15 20:48:58.829998 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:58.829971    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8ff996f-3019-49a3-a67a-89beb08d8676-tmp\") pod \"klusterlet-addon-workmgr-7f5b688c88-krjrp\" (UID: \"b8ff996f-3019-49a3-a67a-89beb08d8676\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5b688c88-krjrp"
Jul 15 20:48:58.830311 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:58.830268    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b1a8d32-166f-4e11-8089-856faaac212d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-86dd496bfc-kzhj5\" (UID: \"2b1a8d32-166f-4e11-8089-856faaac212d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86dd496bfc-kzhj5"
Jul 15 20:48:58.832381 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:58.832360    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2b1a8d32-166f-4e11-8089-856faaac212d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-86dd496bfc-kzhj5\" (UID: \"2b1a8d32-166f-4e11-8089-856faaac212d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86dd496bfc-kzhj5"
Jul 15 20:48:58.832475 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:58.832434    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2b1a8d32-166f-4e11-8089-856faaac212d-hub\") pod \"cluster-proxy-proxy-agent-86dd496bfc-kzhj5\" (UID: \"2b1a8d32-166f-4e11-8089-856faaac212d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86dd496bfc-kzhj5"
Jul 15 20:48:58.832847 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:58.832812    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2b1a8d32-166f-4e11-8089-856faaac212d-ca\") pod \"cluster-proxy-proxy-agent-86dd496bfc-kzhj5\" (UID: \"2b1a8d32-166f-4e11-8089-856faaac212d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86dd496bfc-kzhj5"
Jul 15 20:48:58.832958 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:58.832921    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b8ff996f-3019-49a3-a67a-89beb08d8676-klusterlet-config\") pod \"klusterlet-addon-workmgr-7f5b688c88-krjrp\" (UID: \"b8ff996f-3019-49a3-a67a-89beb08d8676\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5b688c88-krjrp"
Jul 15 20:48:58.833070 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:58.833043    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2b1a8d32-166f-4e11-8089-856faaac212d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-86dd496bfc-kzhj5\" (UID: \"2b1a8d32-166f-4e11-8089-856faaac212d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86dd496bfc-kzhj5"
Jul 15 20:48:58.845213 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:58.845178    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb2jb\" (UniqueName: \"kubernetes.io/projected/2b1a8d32-166f-4e11-8089-856faaac212d-kube-api-access-xb2jb\") pod \"cluster-proxy-proxy-agent-86dd496bfc-kzhj5\" (UID: \"2b1a8d32-166f-4e11-8089-856faaac212d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86dd496bfc-kzhj5"
Jul 15 20:48:58.852264 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:58.852237    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hppn\" (UniqueName: \"kubernetes.io/projected/b8ff996f-3019-49a3-a67a-89beb08d8676-kube-api-access-5hppn\") pod \"klusterlet-addon-workmgr-7f5b688c88-krjrp\" (UID: \"b8ff996f-3019-49a3-a67a-89beb08d8676\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5b688c88-krjrp"
Jul 15 20:48:58.922438 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:58.922399    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:59.010298 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:59.010261    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86dd496bfc-kzhj5"
Jul 15 20:48:59.017253 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:59.017228    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5b688c88-krjrp"
Jul 15 20:48:59.035713 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:59.035679    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:59.123389 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:59.123355    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:59.222246 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:59.222212    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:48:59.325564 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:48:59.325484    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:00.872473 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:00.872434    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:00.972807 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:00.972758    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:01.068658 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:01.068601    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:01.171377 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:01.171355    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:01.188112 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:01.187413    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86dd496bfc-kzhj5"]
Jul 15 20:49:01.190432 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:49:01.190395    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2b1a8d32_166f_4e11_8089_856faaac212d.slice/crio-ed8c48c15967bc174f49a37338239df83444c165a88041cb6864d44e931fea33 WatchSource:0}: Error finding container ed8c48c15967bc174f49a37338239df83444c165a88041cb6864d44e931fea33: Status 404 returned error can't find the container with id ed8c48c15967bc174f49a37338239df83444c165a88041cb6864d44e931fea33
Jul 15 20:49:01.196745 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:01.196711    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ngpfr" event={"ID":"55baaf18-8e1a-4d1b-9411-ed55205cc56c","Type":"ContainerStarted","Data":"ca5b2f45953a48c945b1f4120fe7d77941b8960928d8675b87eebec61a2019e1"}
Jul 15 20:49:01.196996 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:01.196979    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ngpfr"
Jul 15 20:49:01.198222 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:01.198200    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86dd496bfc-kzhj5" event={"ID":"2b1a8d32-166f-4e11-8089-856faaac212d","Type":"ContainerStarted","Data":"ed8c48c15967bc174f49a37338239df83444c165a88041cb6864d44e931fea33"}
Jul 15 20:49:01.201830 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:01.201810    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kwmkr" event={"ID":"075a9740-f176-4bb6-9d0a-9db3ba48fc75","Type":"ContainerStarted","Data":"05bde4089bff3a29398bed2ae7fa47009ed842b77387fb36c5a76198e33e4d64"}
Jul 15 20:49:01.204364 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:01.204343    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5b688c88-krjrp"]
Jul 15 20:49:01.207652 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:49:01.207630    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8ff996f_3019_49a3_a67a_89beb08d8676.slice/crio-265e0724bc573c26a247866d8180d62397dadc74b0a1534f9e9f176fb9e0b037 WatchSource:0}: Error finding container 265e0724bc573c26a247866d8180d62397dadc74b0a1534f9e9f176fb9e0b037: Status 404 returned error can't find the container with id 265e0724bc573c26a247866d8180d62397dadc74b0a1534f9e9f176fb9e0b037
Jul 15 20:49:01.218032 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:01.217978    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ngpfr" podStartSLOduration=66.747742631 podStartE2EDuration="1m13.217960491s" totalImagesPullingTime="6.47021786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:47:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:49:01.217552262 +0000 UTC m=+74.366590321" watchObservedRunningTime="2026-07-15 20:49:01.217960491 +0000 UTC m=+74.366998549"
Jul 15 20:49:01.237005 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:01.236951    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zzw8h" podStartSLOduration=66.954003173 podStartE2EDuration="1m13.236935422s" totalImagesPullingTime="6.282932249s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 20:47:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:49:01.235917212 +0000 UTC m=+74.384955290" watchObservedRunningTime="2026-07-15 20:49:01.236935422 +0000 UTC m=+74.385973479"
Jul 15 20:49:01.255726 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:01.255673    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kwmkr" podStartSLOduration=17.327010289 podStartE2EDuration="32.255657397s" totalImagesPullingTime="14.928647108s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 20:48:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:49:01.25495134 +0000 UTC m=+74.403989398" watchObservedRunningTime="2026-07-15 20:49:01.255657397 +0000 UTC m=+74.404695453"
Jul 15 20:49:01.647965 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:01.647930    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:01.749390 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:01.749350    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:01.870781 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:01.870748    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:01.981728 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:01.981694    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:02.082188 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:02.082156    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:02.180165 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:02.180124    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:02.206570 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:02.206531    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5b688c88-krjrp" event={"ID":"b8ff996f-3019-49a3-a67a-89beb08d8676","Type":"ContainerStarted","Data":"265e0724bc573c26a247866d8180d62397dadc74b0a1534f9e9f176fb9e0b037"}
Jul 15 20:49:02.208573 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:02.208498    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zzw8h" event={"ID":"b9d9267e-0c23-42b2-bb48-55ed14050fff","Type":"ContainerStarted","Data":"882ca77456b556607af77c0f833e2331c74eccf892392bc0c4f58e56b558623d"}
Jul 15 20:49:02.390982 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:02.390871    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:02.564164 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:02.564120    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:02.639085 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:02.639037    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:02.975790 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:02.975748    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:03.069760 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:03.069722    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:03.756349 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:03.756317    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:03.883145 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:03.883031    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:03.984019 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:03.983979    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:04.049489 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:04.049359    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:04.312580 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:04.312495    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:04.571053 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:04.570959    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:05.080483 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:05.080392    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:05.182992 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:05.181840    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:05.263014 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:05.262969    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:06.228668 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:06.228259    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5b688c88-krjrp" event={"ID":"b8ff996f-3019-49a3-a67a-89beb08d8676","Type":"ContainerStarted","Data":"6e9ab75d3e83b9705d48a2aa01f04506900732046f38482f870cc94710aaddda"}
Jul 15 20:49:06.228668 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:06.228551    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5b688c88-krjrp"
Jul 15 20:49:06.230307 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:06.230287    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5b688c88-krjrp"
Jul 15 20:49:06.262605 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:06.262525    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5b688c88-krjrp" podStartSLOduration=3.855499091 podStartE2EDuration="8.26250214s" totalImagesPullingTime="4.407003049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:48:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:49:06.245919622 +0000 UTC m=+79.394957674" watchObservedRunningTime="2026-07-15 20:49:06.26250214 +0000 UTC m=+79.411540197"
Jul 15 20:49:07.774075 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:07.774041    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:07.895271 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:07.895238    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:07.994466 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:07.994435    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:08.236816 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:08.236776    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86dd496bfc-kzhj5" event={"ID":"2b1a8d32-166f-4e11-8089-856faaac212d","Type":"ContainerStarted","Data":"8d830096dc540ecf432033e4e3814628ff56ca27df30055a4d5e472285b51ef1"}
Jul 15 20:49:08.236816 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:08.236818    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86dd496bfc-kzhj5" event={"ID":"2b1a8d32-166f-4e11-8089-856faaac212d","Type":"ContainerStarted","Data":"f706a4a2bc3c37c1cb36040a318ae2858fa2e15f9d48e9a3072ae03b23fa6f9d"}
Jul 15 20:49:08.237078 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:08.236830    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86dd496bfc-kzhj5" event={"ID":"2b1a8d32-166f-4e11-8089-856faaac212d","Type":"ContainerStarted","Data":"15fb4230adb1692269078bfade83c8224d2b5d1930d027947eb7bb5a6704daca"}
Jul 15 20:49:08.262308 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:08.262261    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86dd496bfc-kzhj5" podStartSLOduration=3.676981194 podStartE2EDuration="10.26224687s" totalImagesPullingTime="6.585265676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:48:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:49:08.261592148 +0000 UTC m=+81.410630228" watchObservedRunningTime="2026-07-15 20:49:08.26224687 +0000 UTC m=+81.411284927"
Jul 15 20:49:09.645918 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:09.645866    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:09.781954 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:09.781917    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:09.879134 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:09.879102    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:10.256253 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:10.256214    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:10.520482 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:10.520387    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:10.549961 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:10.549926    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tjq5h"]
Jul 15 20:49:10.556825 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:10.556798    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tjq5h"
Jul 15 20:49:10.559914 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:10.559872    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 15 20:49:10.560028 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:10.559923    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 15 20:49:10.560322 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:10.560302    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 15 20:49:10.560410 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:10.560339    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 15 20:49:10.560410 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:10.560305    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2qz5x\""
Jul 15 20:49:10.560523 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:10.560358    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 15 20:49:10.561030 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:10.561014    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 15 20:49:10.620845 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:10.620811    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:10.626873 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:10.626848    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84bc5eba-a859-45de-8e04-2a65286f40f9-sys\") pod \"node-exporter-tjq5h\" (UID: \"84bc5eba-a859-45de-8e04-2a65286f40f9\") " pod="openshift-monitoring/node-exporter-tjq5h"
Jul 15 20:49:10.627063 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:10.626888    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drfcm\" (UniqueName: \"kubernetes.io/projected/84bc5eba-a859-45de-8e04-2a65286f40f9-kube-api-access-drfcm\") pod \"node-exporter-tjq5h\" (UID: \"84bc5eba-a859-45de-8e04-2a65286f40f9\") " pod="openshift-monitoring/node-exporter-tjq5h"
Jul 15 20:49:10.627063 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:10.626965    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84bc5eba-a859-45de-8e04-2a65286f40f9-metrics-client-ca\") pod \"node-exporter-tjq5h\" (UID: \"84bc5eba-a859-45de-8e04-2a65286f40f9\") " pod="openshift-monitoring/node-exporter-tjq5h"
Jul 15 20:49:10.627063 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:10.627023    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/84bc5eba-a859-45de-8e04-2a65286f40f9-node-exporter-tls\") pod \"node-exporter-tjq5h\" (UID: \"84bc5eba-a859-45de-8e04-2a65286f40f9\") " pod="openshift-monitoring/node-exporter-tjq5h"
Jul 15 20:49:10.627191 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:10.627071    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/84bc5eba-a859-45de-8e04-2a65286f40f9-node-exporter-wtmp\") pod \"node-exporter-tjq5h\" (UID: \"84bc5eba-a859-45de-8e04-2a65286f40f9\") " pod="openshift-monitoring/node-exporter-tjq5h"
Jul 15 20:49:10.627191 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:10.627097    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/84bc5eba-a859-45de-8e04-2a65286f40f9-root\") pod \"node-exporter-tjq5h\" (UID: \"84bc5eba-a859-45de-8e04-2a65286f40f9\") " pod="openshift-monitoring/node-exporter-tjq5h"
Jul 15 20:49:10.627191 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:10.627159    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/84bc5eba-a859-45de-8e04-2a65286f40f9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tjq5h\" (UID: \"84bc5eba-a859-45de-8e04-2a65286f40f9\") " pod="openshift-monitoring/node-exporter-tjq5h"
Jul 15 20:49:10.627191 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:10.627183    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/84bc5eba-a859-45de-8e04-2a65286f40f9-node-exporter-accelerators-collector-config\") pod \"node-exporter-tjq5h\" (UID: \"84bc5eba-a859-45de-8e04-2a65286f40f9\") " pod="openshift-monitoring/node-exporter-tjq5h"
Jul 15 20:49:10.627324 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:10.627217    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/84bc5eba-a859-45de-8e04-2a65286f40f9-node-exporter-textfile\") pod \"node-exporter-tjq5h\" (UID: \"84bc5eba-a859-45de-8e04-2a65286f40f9\") " pod="openshift-monitoring/node-exporter-tjq5h"
Jul 15 20:49:10.728511 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:10.728462    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/84bc5eba-a859-45de-8e04-2a65286f40f9-node-exporter-wtmp\") pod \"node-exporter-tjq5h\" (UID: \"84bc5eba-a859-45de-8e04-2a65286f40f9\") " pod="openshift-monitoring/node-exporter-tjq5h"
Jul 15 20:49:10.728982 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:10.728516    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/84bc5eba-a859-45de-8e04-2a65286f40f9-root\") pod \"node-exporter-tjq5h\" (UID: \"84bc5eba-a859-45de-8e04-2a65286f40f9\") " pod="openshift-monitoring/node-exporter-tjq5h"
Jul 15 20:49:10.728982 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:10.728566    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/84bc5eba-a859-45de-8e04-2a65286f40f9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tjq5h\" (UID: \"84bc5eba-a859-45de-8e04-2a65286f40f9\") " pod="openshift-monitoring/node-exporter-tjq5h"
Jul 15 20:49:10.728982 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:10.728593    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/84bc5eba-a859-45de-8e04-2a65286f40f9-node-exporter-accelerators-collector-config\") pod \"node-exporter-tjq5h\" (UID: \"84bc5eba-a859-45de-8e04-2a65286f40f9\") " pod="openshift-monitoring/node-exporter-tjq5h"
Jul 15 20:49:10.728982 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:10.728623    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/84bc5eba-a859-45de-8e04-2a65286f40f9-node-exporter-textfile\") pod \"node-exporter-tjq5h\" (UID: \"84bc5eba-a859-45de-8e04-2a65286f40f9\") " pod="openshift-monitoring/node-exporter-tjq5h"
Jul 15 20:49:10.728982 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:10.728639    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/84bc5eba-a859-45de-8e04-2a65286f40f9-node-exporter-wtmp\") pod \"node-exporter-tjq5h\" (UID: \"84bc5eba-a859-45de-8e04-2a65286f40f9\") " pod="openshift-monitoring/node-exporter-tjq5h"
Jul 15 20:49:10.728982 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:10.728656    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84bc5eba-a859-45de-8e04-2a65286f40f9-sys\") pod \"node-exporter-tjq5h\" (UID: \"84bc5eba-a859-45de-8e04-2a65286f40f9\") " pod="openshift-monitoring/node-exporter-tjq5h"
Jul 15 20:49:10.728982 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:10.728704    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84bc5eba-a859-45de-8e04-2a65286f40f9-sys\") pod \"node-exporter-tjq5h\" (UID: \"84bc5eba-a859-45de-8e04-2a65286f40f9\") " pod="openshift-monitoring/node-exporter-tjq5h"
Jul 15 20:49:10.728982 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:10.728702    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drfcm\" (UniqueName: \"kubernetes.io/projected/84bc5eba-a859-45de-8e04-2a65286f40f9-kube-api-access-drfcm\") pod \"node-exporter-tjq5h\" (UID: \"84bc5eba-a859-45de-8e04-2a65286f40f9\") " pod="openshift-monitoring/node-exporter-tjq5h"
Jul 15 20:49:10.728982 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:10.728640    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/84bc5eba-a859-45de-8e04-2a65286f40f9-root\") pod \"node-exporter-tjq5h\" (UID: \"84bc5eba-a859-45de-8e04-2a65286f40f9\") " pod="openshift-monitoring/node-exporter-tjq5h"
Jul 15 20:49:10.728982 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:10.728822    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84bc5eba-a859-45de-8e04-2a65286f40f9-metrics-client-ca\") pod \"node-exporter-tjq5h\" (UID: \"84bc5eba-a859-45de-8e04-2a65286f40f9\") " pod="openshift-monitoring/node-exporter-tjq5h"
Jul 15 20:49:10.728982 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:10.728875    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/84bc5eba-a859-45de-8e04-2a65286f40f9-node-exporter-tls\") pod \"node-exporter-tjq5h\" (UID: \"84bc5eba-a859-45de-8e04-2a65286f40f9\") " pod="openshift-monitoring/node-exporter-tjq5h"
Jul 15 20:49:10.729579 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:10.729004    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/84bc5eba-a859-45de-8e04-2a65286f40f9-node-exporter-textfile\") pod \"node-exporter-tjq5h\" (UID: \"84bc5eba-a859-45de-8e04-2a65286f40f9\") " pod="openshift-monitoring/node-exporter-tjq5h"
Jul 15 20:49:10.729579 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:49:10.729160    2566 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 15 20:49:10.729579 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:49:10.729234    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84bc5eba-a859-45de-8e04-2a65286f40f9-node-exporter-tls podName:84bc5eba-a859-45de-8e04-2a65286f40f9 nodeName:}" failed. No retries permitted until 2026-07-15 20:49:11.229215003 +0000 UTC m=+84.378253043 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/84bc5eba-a859-45de-8e04-2a65286f40f9-node-exporter-tls") pod "node-exporter-tjq5h" (UID: "84bc5eba-a859-45de-8e04-2a65286f40f9") : secret "node-exporter-tls" not found
Jul 15 20:49:10.729579 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:10.729308    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/84bc5eba-a859-45de-8e04-2a65286f40f9-node-exporter-accelerators-collector-config\") pod \"node-exporter-tjq5h\" (UID: \"84bc5eba-a859-45de-8e04-2a65286f40f9\") " pod="openshift-monitoring/node-exporter-tjq5h"
Jul 15 20:49:10.729579 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:10.729473    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84bc5eba-a859-45de-8e04-2a65286f40f9-metrics-client-ca\") pod \"node-exporter-tjq5h\" (UID: \"84bc5eba-a859-45de-8e04-2a65286f40f9\") " pod="openshift-monitoring/node-exporter-tjq5h"
Jul 15 20:49:10.730916 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:10.730882    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/84bc5eba-a859-45de-8e04-2a65286f40f9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tjq5h\" (UID: \"84bc5eba-a859-45de-8e04-2a65286f40f9\") " pod="openshift-monitoring/node-exporter-tjq5h"
Jul 15 20:49:10.739325 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:10.739300    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drfcm\" (UniqueName: \"kubernetes.io/projected/84bc5eba-a859-45de-8e04-2a65286f40f9-kube-api-access-drfcm\") pod \"node-exporter-tjq5h\" (UID: \"84bc5eba-a859-45de-8e04-2a65286f40f9\") " pod="openshift-monitoring/node-exporter-tjq5h"
Jul 15 20:49:10.778282 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:10.778185    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:10.876589 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:10.876553    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:11.074738 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:11.074658    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:11.188296 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:11.188258    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:11.234505 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:11.234474    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/84bc5eba-a859-45de-8e04-2a65286f40f9-node-exporter-tls\") pod \"node-exporter-tjq5h\" (UID: \"84bc5eba-a859-45de-8e04-2a65286f40f9\") " pod="openshift-monitoring/node-exporter-tjq5h"
Jul 15 20:49:11.236695 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:11.236670    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/84bc5eba-a859-45de-8e04-2a65286f40f9-node-exporter-tls\") pod \"node-exporter-tjq5h\" (UID: \"84bc5eba-a859-45de-8e04-2a65286f40f9\") " pod="openshift-monitoring/node-exporter-tjq5h"
Jul 15 20:49:11.286397 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:11.286353    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:11.468049 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:11.468021    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tjq5h"
Jul 15 20:49:11.483867 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:49:11.483829    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84bc5eba_a859_45de_8e04_2a65286f40f9.slice/crio-7a1e9403f73af329c1e1b8948ec534f50d659089836684e3de12522214be396a WatchSource:0}: Error finding container 7a1e9403f73af329c1e1b8948ec534f50d659089836684e3de12522214be396a: Status 404 returned error can't find the container with id 7a1e9403f73af329c1e1b8948ec534f50d659089836684e3de12522214be396a
Jul 15 20:49:11.546552 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:11.546520    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:11.670770 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:11.670735    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:11.894625 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:11.894532    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:12.251918 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:12.251871    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tjq5h" event={"ID":"84bc5eba-a859-45de-8e04-2a65286f40f9","Type":"ContainerStarted","Data":"7a1e9403f73af329c1e1b8948ec534f50d659089836684e3de12522214be396a"}
Jul 15 20:49:12.979594 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:12.979549    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:13.091242 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:13.091208    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:13.236344 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:13.236248    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:13.256420 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:13.256386    2566 generic.go:356] "Generic (PLEG): container finished" podID="84bc5eba-a859-45de-8e04-2a65286f40f9" containerID="3588d430f45afe8f169d4896e7d64f725305e5e9d1f89763338ba861a202e2d1" exitCode=0
Jul 15 20:49:13.256577 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:13.256440    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tjq5h" event={"ID":"84bc5eba-a859-45de-8e04-2a65286f40f9","Type":"ContainerDied","Data":"3588d430f45afe8f169d4896e7d64f725305e5e9d1f89763338ba861a202e2d1"}
Jul 15 20:49:13.340707 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:13.340684    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:13.449806 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:13.449751    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:13.564414 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:13.564324    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:13.961461 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:13.961430    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:14.064510 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:14.064483    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:14.261536 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:14.261455    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tjq5h" event={"ID":"84bc5eba-a859-45de-8e04-2a65286f40f9","Type":"ContainerStarted","Data":"81c615de0d92d30ea52bf4218329665a81fb2bbcef32083ccfe53003e98bff4f"}
Jul 15 20:49:14.261536 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:14.261510    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tjq5h" event={"ID":"84bc5eba-a859-45de-8e04-2a65286f40f9","Type":"ContainerStarted","Data":"f3d7d993fd38c9fffec0e206d7b2e142cbb4f2214f10e7142b2690a3a1c34447"}
Jul 15 20:49:14.285103 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:14.285050    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tjq5h" podStartSLOduration=3.224157679 podStartE2EDuration="4.285034047s" totalImagesPullingTime="998.575554ms" totalInitContainerRuntime="62.300814ms" isStatefulPod=true podCreationTimestamp="2026-07-15 20:49:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:49:14.2826547 +0000 UTC m=+87.431692756" watchObservedRunningTime="2026-07-15 20:49:14.285034047 +0000 UTC m=+87.434072104"
Jul 15 20:49:15.310882 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:15.310854    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:15.403731 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:15.403703    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:15.690293 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:15.690211    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:16.413303 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:16.413271    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:16.581619 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:16.581588    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:16.699505 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:16.699465    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:16.867474 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:16.867424    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:16.870184 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:16.870163    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 15 20:49:16.875761 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:16.875746    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:16.878926 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:16.878887    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 15 20:49:16.878926 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:16.878888    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 15 20:49:16.879085 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:16.878943    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-471ecd56nuucl\""
Jul 15 20:49:16.879085 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:16.878927    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 15 20:49:16.879085 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:16.878927    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 15 20:49:16.879085 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:16.879037    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 15 20:49:16.879085 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:16.879045    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 15 20:49:16.879380 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:16.879361    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 15 20:49:16.879494 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:16.879384    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 15 20:49:16.879566 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:16.879553    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 15 20:49:16.880223 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:16.880205    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 15 20:49:16.880316 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:16.880206    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 15 20:49:16.880566 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:16.880547    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 15 20:49:16.880566 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:16.880555    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 15 20:49:16.880696 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:16.880615    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7qhn7\""
Jul 15 20:49:16.885450 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:16.885430    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 15 20:49:16.887808 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:16.887793    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 15 20:49:16.896996 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:16.896973    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 15 20:49:16.980271 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:16.980160    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:16.980271 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:16.980220    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aaa43101-957c-4064-92db-ba62aabdd8d9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:16.980271 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:16.980259    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:16.980522 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:16.980284    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:16.980522 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:16.980314    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/aaa43101-957c-4064-92db-ba62aabdd8d9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:16.980522 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:16.980339    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:16.980522 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:16.980359    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aaa43101-957c-4064-92db-ba62aabdd8d9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:16.980522 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:16.980381    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:16.980522 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:16.980405    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:16.980522 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:16.980438    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:16.980522 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:16.980465    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aaa43101-957c-4064-92db-ba62aabdd8d9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:16.980522 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:16.980490    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-web-config\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:16.980522 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:16.980519    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/aaa43101-957c-4064-92db-ba62aabdd8d9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:16.980886 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:16.980564    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2b4j\" (UniqueName: \"kubernetes.io/projected/aaa43101-957c-4064-92db-ba62aabdd8d9-kube-api-access-s2b4j\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:16.980886 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:16.980598    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aaa43101-957c-4064-92db-ba62aabdd8d9-config-out\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:16.980886 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:16.980616    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-config\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:16.980886 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:16.980633    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aaa43101-957c-4064-92db-ba62aabdd8d9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:16.980886 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:16.980652    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/aaa43101-957c-4064-92db-ba62aabdd8d9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:16.980886 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:16.980669    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aaa43101-957c-4064-92db-ba62aabdd8d9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:16.980886 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:16.980740    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/aaa43101-957c-4064-92db-ba62aabdd8d9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:17.011050 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:17.011020    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:17.081584 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:17.081551    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:17.081584 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:17.081587    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:17.081789 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:17.081609    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/aaa43101-957c-4064-92db-ba62aabdd8d9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:17.081789 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:17.081725    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:17.081789 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:17.081763    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aaa43101-957c-4064-92db-ba62aabdd8d9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:17.081982 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:17.081793    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:17.081982 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:17.081817    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:17.081982 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:17.081853    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:17.081982 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:17.081877    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aaa43101-957c-4064-92db-ba62aabdd8d9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:17.081982 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:17.081929    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-web-config\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:17.081982 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:17.081958    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/aaa43101-957c-4064-92db-ba62aabdd8d9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:17.082277 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:17.081992    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2b4j\" (UniqueName: \"kubernetes.io/projected/aaa43101-957c-4064-92db-ba62aabdd8d9-kube-api-access-s2b4j\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:17.082277 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:17.082028    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aaa43101-957c-4064-92db-ba62aabdd8d9-config-out\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:17.082277 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:17.082049    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-config\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:17.082277 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:17.082084    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aaa43101-957c-4064-92db-ba62aabdd8d9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:17.082277 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:17.082113    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/aaa43101-957c-4064-92db-ba62aabdd8d9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:17.082277 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:17.082140    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aaa43101-957c-4064-92db-ba62aabdd8d9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:17.082277 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:17.082183    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/aaa43101-957c-4064-92db-ba62aabdd8d9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:17.082277 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:17.082226    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:17.082277 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:17.082253    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aaa43101-957c-4064-92db-ba62aabdd8d9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:17.082729 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:17.082393    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/aaa43101-957c-4064-92db-ba62aabdd8d9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:17.082729 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:17.082489    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aaa43101-957c-4064-92db-ba62aabdd8d9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:17.082938 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:17.082913    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/aaa43101-957c-4064-92db-ba62aabdd8d9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:17.083721 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:17.083030    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aaa43101-957c-4064-92db-ba62aabdd8d9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:17.083838 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:17.083299    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/aaa43101-957c-4064-92db-ba62aabdd8d9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:17.084290 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:17.083498    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aaa43101-957c-4064-92db-ba62aabdd8d9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:17.084855 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:17.084811    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:17.085050 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:17.085025    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aaa43101-957c-4064-92db-ba62aabdd8d9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:17.085692 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:17.085664    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:17.086168 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:17.086142    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:17.086255 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:17.086220    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:17.086862 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:17.086416    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aaa43101-957c-4064-92db-ba62aabdd8d9-config-out\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:17.086862 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:17.086823    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:17.087033 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:17.086982    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-web-config\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:17.087509 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:17.087489    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:17.087633 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:17.087618    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-config\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:17.087691 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:17.087671    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aaa43101-957c-4064-92db-ba62aabdd8d9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:17.087858 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:17.087835    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:17.087966 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:17.087949    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/aaa43101-957c-4064-92db-ba62aabdd8d9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:17.125247 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:17.125214    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2b4j\" (UniqueName: \"kubernetes.io/projected/aaa43101-957c-4064-92db-ba62aabdd8d9-kube-api-access-s2b4j\") pod \"prometheus-k8s-0\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:17.185413 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:17.185374    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:17.328225 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:17.328194    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 15 20:49:17.331097 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:49:17.331068    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaa43101_957c_4064_92db_ba62aabdd8d9.slice/crio-37eb16d6d7bf471589d803aadb3122bf9fff819995aa3a76258c02d1aaa1b8af WatchSource:0}: Error finding container 37eb16d6d7bf471589d803aadb3122bf9fff819995aa3a76258c02d1aaa1b8af: Status 404 returned error can't find the container with id 37eb16d6d7bf471589d803aadb3122bf9fff819995aa3a76258c02d1aaa1b8af
Jul 15 20:49:17.732069 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:17.732036    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:17.869070 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:17.869028    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:18.277747 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:18.277708    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aaa43101-957c-4064-92db-ba62aabdd8d9","Type":"ContainerStarted","Data":"37eb16d6d7bf471589d803aadb3122bf9fff819995aa3a76258c02d1aaa1b8af"}
Jul 15 20:49:19.282936 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:19.282879    2566 generic.go:356] "Generic (PLEG): container finished" podID="aaa43101-957c-4064-92db-ba62aabdd8d9" containerID="65dad387b1d38b98aad7bffbc9fab2c6014a9ccde5afaf16fa25a277ac084064" exitCode=0
Jul 15 20:49:19.282936 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:19.282924    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aaa43101-957c-4064-92db-ba62aabdd8d9","Type":"ContainerDied","Data":"65dad387b1d38b98aad7bffbc9fab2c6014a9ccde5afaf16fa25a277ac084064"}
Jul 15 20:49:20.180376 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:20.180332    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:20.259083 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:20.259050    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:23.153866 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:23.153776    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:23.255077 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:23.255041    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:23.302220 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:23.302180    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aaa43101-957c-4064-92db-ba62aabdd8d9","Type":"ContainerStarted","Data":"6c426b6ad12dbcd77f7c71c2742451f837b4ec1ef415d58b2cedd652cb172beb"}
Jul 15 20:49:23.302392 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:23.302249    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aaa43101-957c-4064-92db-ba62aabdd8d9","Type":"ContainerStarted","Data":"c0ebfdc78c0faab0b9019b484d760694c723fd61f20348fa38b76443158c6ade"}
Jul 15 20:49:23.631956 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:23.631906    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:23.708625 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:23.708585    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:23.804629 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:23.804596    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:25.312702 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:25.312665    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aaa43101-957c-4064-92db-ba62aabdd8d9","Type":"ContainerStarted","Data":"5e13a4a9f233db12707d55073929ff3354e6fbc2c0ea94a0f6a52e19c7c8fa32"}
Jul 15 20:49:25.313146 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:25.312731    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aaa43101-957c-4064-92db-ba62aabdd8d9","Type":"ContainerStarted","Data":"85fe3276d3cc66b1ce64bbf04e29349355633893d8c54243e7471cb8b49b30dc"}
Jul 15 20:49:25.313146 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:25.312759    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aaa43101-957c-4064-92db-ba62aabdd8d9","Type":"ContainerStarted","Data":"b799ddcc83fc373e734d61665661ca522f32ca5440cd742fab7395802a93d255"}
Jul 15 20:49:25.313920 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:25.313404    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aaa43101-957c-4064-92db-ba62aabdd8d9","Type":"ContainerStarted","Data":"96f71b369196aef28ae04776dbbb4c281b3affffb3a304d926bfdeba9329621c"}
Jul 15 20:49:25.354058 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:25.354002    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.108068236 podStartE2EDuration="9.353984205s" totalImagesPullingTime="6.227304686s" totalInitContainerRuntime="18.611283ms" isStatefulPod=false podCreationTimestamp="2026-07-15 20:49:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:49:25.351682133 +0000 UTC m=+98.500720226" watchObservedRunningTime="2026-07-15 20:49:25.353984205 +0000 UTC m=+98.503022283"
Jul 15 20:49:26.754057 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:26.754022    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:26.908419 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:26.908387    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:27.185861 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:27.185776    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:49:27.394184 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:27.394155    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:27.500140 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:27.500100    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:31.771826 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:31.771785    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:31.964302 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:31.964271    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:32.211529 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:32.211500    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ngpfr"
Jul 15 20:49:34.785438 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:34.785398    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:34.936744 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:34.936712    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:46.688924 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:46.688824    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:46.788906 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:46.788835    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:47.066858 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:47.066827    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:49:47.189641 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:49:47.189603    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:50:11.819834 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:11.819795    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:50:17.186603 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:17.186565    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:17.206773 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:17.206741    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:17.516112 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:17.516081    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:29.942162 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:29.942127    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:50:30.048946 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:30.048890    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:50:35.300591 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.300555    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 15 20:50:35.301037 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.300884    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="aaa43101-957c-4064-92db-ba62aabdd8d9" containerName="prometheus" containerID="cri-o://c0ebfdc78c0faab0b9019b484d760694c723fd61f20348fa38b76443158c6ade" gracePeriod=600
Jul 15 20:50:35.301037 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.300941    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="aaa43101-957c-4064-92db-ba62aabdd8d9" containerName="kube-rbac-proxy" containerID="cri-o://85fe3276d3cc66b1ce64bbf04e29349355633893d8c54243e7471cb8b49b30dc" gracePeriod=600
Jul 15 20:50:35.301037 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.300942    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="aaa43101-957c-4064-92db-ba62aabdd8d9" containerName="thanos-sidecar" containerID="cri-o://96f71b369196aef28ae04776dbbb4c281b3affffb3a304d926bfdeba9329621c" gracePeriod=600
Jul 15 20:50:35.301037 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.300959    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="aaa43101-957c-4064-92db-ba62aabdd8d9" containerName="config-reloader" containerID="cri-o://6c426b6ad12dbcd77f7c71c2742451f837b4ec1ef415d58b2cedd652cb172beb" gracePeriod=600
Jul 15 20:50:35.301037 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.300938    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="aaa43101-957c-4064-92db-ba62aabdd8d9" containerName="kube-rbac-proxy-thanos" containerID="cri-o://5e13a4a9f233db12707d55073929ff3354e6fbc2c0ea94a0f6a52e19c7c8fa32" gracePeriod=600
Jul 15 20:50:35.301314 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.300990    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="aaa43101-957c-4064-92db-ba62aabdd8d9" containerName="kube-rbac-proxy-web" containerID="cri-o://b799ddcc83fc373e734d61665661ca522f32ca5440cd742fab7395802a93d255" gracePeriod=600
Jul 15 20:50:35.548044 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.548021    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:35.562194 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.562133    2566 generic.go:356] "Generic (PLEG): container finished" podID="aaa43101-957c-4064-92db-ba62aabdd8d9" containerID="5e13a4a9f233db12707d55073929ff3354e6fbc2c0ea94a0f6a52e19c7c8fa32" exitCode=0
Jul 15 20:50:35.562194 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.562158    2566 generic.go:356] "Generic (PLEG): container finished" podID="aaa43101-957c-4064-92db-ba62aabdd8d9" containerID="85fe3276d3cc66b1ce64bbf04e29349355633893d8c54243e7471cb8b49b30dc" exitCode=0
Jul 15 20:50:35.562194 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.562165    2566 generic.go:356] "Generic (PLEG): container finished" podID="aaa43101-957c-4064-92db-ba62aabdd8d9" containerID="b799ddcc83fc373e734d61665661ca522f32ca5440cd742fab7395802a93d255" exitCode=0
Jul 15 20:50:35.562194 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.562162    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aaa43101-957c-4064-92db-ba62aabdd8d9","Type":"ContainerDied","Data":"5e13a4a9f233db12707d55073929ff3354e6fbc2c0ea94a0f6a52e19c7c8fa32"}
Jul 15 20:50:35.562416 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.562207    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:35.562416 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.562229    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aaa43101-957c-4064-92db-ba62aabdd8d9","Type":"ContainerDied","Data":"85fe3276d3cc66b1ce64bbf04e29349355633893d8c54243e7471cb8b49b30dc"}
Jul 15 20:50:35.562416 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.562171    2566 generic.go:356] "Generic (PLEG): container finished" podID="aaa43101-957c-4064-92db-ba62aabdd8d9" containerID="96f71b369196aef28ae04776dbbb4c281b3affffb3a304d926bfdeba9329621c" exitCode=0
Jul 15 20:50:35.562416 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.562254    2566 generic.go:356] "Generic (PLEG): container finished" podID="aaa43101-957c-4064-92db-ba62aabdd8d9" containerID="6c426b6ad12dbcd77f7c71c2742451f837b4ec1ef415d58b2cedd652cb172beb" exitCode=0
Jul 15 20:50:35.562416 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.562259    2566 generic.go:356] "Generic (PLEG): container finished" podID="aaa43101-957c-4064-92db-ba62aabdd8d9" containerID="c0ebfdc78c0faab0b9019b484d760694c723fd61f20348fa38b76443158c6ade" exitCode=0
Jul 15 20:50:35.562416 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.562280    2566 scope.go:117] "RemoveContainer" containerID="5e13a4a9f233db12707d55073929ff3354e6fbc2c0ea94a0f6a52e19c7c8fa32"
Jul 15 20:50:35.562416 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.562259    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aaa43101-957c-4064-92db-ba62aabdd8d9","Type":"ContainerDied","Data":"b799ddcc83fc373e734d61665661ca522f32ca5440cd742fab7395802a93d255"}
Jul 15 20:50:35.562416 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.562354    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aaa43101-957c-4064-92db-ba62aabdd8d9","Type":"ContainerDied","Data":"96f71b369196aef28ae04776dbbb4c281b3affffb3a304d926bfdeba9329621c"}
Jul 15 20:50:35.562416 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.562382    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aaa43101-957c-4064-92db-ba62aabdd8d9","Type":"ContainerDied","Data":"6c426b6ad12dbcd77f7c71c2742451f837b4ec1ef415d58b2cedd652cb172beb"}
Jul 15 20:50:35.562761 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.562425    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aaa43101-957c-4064-92db-ba62aabdd8d9","Type":"ContainerDied","Data":"c0ebfdc78c0faab0b9019b484d760694c723fd61f20348fa38b76443158c6ade"}
Jul 15 20:50:35.562761 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.562458    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aaa43101-957c-4064-92db-ba62aabdd8d9","Type":"ContainerDied","Data":"37eb16d6d7bf471589d803aadb3122bf9fff819995aa3a76258c02d1aaa1b8af"}
Jul 15 20:50:35.569426 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.569409    2566 scope.go:117] "RemoveContainer" containerID="85fe3276d3cc66b1ce64bbf04e29349355633893d8c54243e7471cb8b49b30dc"
Jul 15 20:50:35.576201 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.576184    2566 scope.go:117] "RemoveContainer" containerID="b799ddcc83fc373e734d61665661ca522f32ca5440cd742fab7395802a93d255"
Jul 15 20:50:35.583009 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.582987    2566 scope.go:117] "RemoveContainer" containerID="96f71b369196aef28ae04776dbbb4c281b3affffb3a304d926bfdeba9329621c"
Jul 15 20:50:35.589941 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.589915    2566 scope.go:117] "RemoveContainer" containerID="6c426b6ad12dbcd77f7c71c2742451f837b4ec1ef415d58b2cedd652cb172beb"
Jul 15 20:50:35.597304 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.597283    2566 scope.go:117] "RemoveContainer" containerID="c0ebfdc78c0faab0b9019b484d760694c723fd61f20348fa38b76443158c6ade"
Jul 15 20:50:35.604200 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.604181    2566 scope.go:117] "RemoveContainer" containerID="65dad387b1d38b98aad7bffbc9fab2c6014a9ccde5afaf16fa25a277ac084064"
Jul 15 20:50:35.611668 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.611644    2566 scope.go:117] "RemoveContainer" containerID="5e13a4a9f233db12707d55073929ff3354e6fbc2c0ea94a0f6a52e19c7c8fa32"
Jul 15 20:50:35.611978 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:50:35.611957    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e13a4a9f233db12707d55073929ff3354e6fbc2c0ea94a0f6a52e19c7c8fa32\": container with ID starting with 5e13a4a9f233db12707d55073929ff3354e6fbc2c0ea94a0f6a52e19c7c8fa32 not found: ID does not exist" containerID="5e13a4a9f233db12707d55073929ff3354e6fbc2c0ea94a0f6a52e19c7c8fa32"
Jul 15 20:50:35.612051 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.611986    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e13a4a9f233db12707d55073929ff3354e6fbc2c0ea94a0f6a52e19c7c8fa32"} err="failed to get container status \"5e13a4a9f233db12707d55073929ff3354e6fbc2c0ea94a0f6a52e19c7c8fa32\": rpc error: code = NotFound desc = could not find container \"5e13a4a9f233db12707d55073929ff3354e6fbc2c0ea94a0f6a52e19c7c8fa32\": container with ID starting with 5e13a4a9f233db12707d55073929ff3354e6fbc2c0ea94a0f6a52e19c7c8fa32 not found: ID does not exist"
Jul 15 20:50:35.612051 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.612021    2566 scope.go:117] "RemoveContainer" containerID="85fe3276d3cc66b1ce64bbf04e29349355633893d8c54243e7471cb8b49b30dc"
Jul 15 20:50:35.612248 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:50:35.612234    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85fe3276d3cc66b1ce64bbf04e29349355633893d8c54243e7471cb8b49b30dc\": container with ID starting with 85fe3276d3cc66b1ce64bbf04e29349355633893d8c54243e7471cb8b49b30dc not found: ID does not exist" containerID="85fe3276d3cc66b1ce64bbf04e29349355633893d8c54243e7471cb8b49b30dc"
Jul 15 20:50:35.612290 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.612252    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85fe3276d3cc66b1ce64bbf04e29349355633893d8c54243e7471cb8b49b30dc"} err="failed to get container status \"85fe3276d3cc66b1ce64bbf04e29349355633893d8c54243e7471cb8b49b30dc\": rpc error: code = NotFound desc = could not find container \"85fe3276d3cc66b1ce64bbf04e29349355633893d8c54243e7471cb8b49b30dc\": container with ID starting with 85fe3276d3cc66b1ce64bbf04e29349355633893d8c54243e7471cb8b49b30dc not found: ID does not exist"
Jul 15 20:50:35.612290 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.612265    2566 scope.go:117] "RemoveContainer" containerID="b799ddcc83fc373e734d61665661ca522f32ca5440cd742fab7395802a93d255"
Jul 15 20:50:35.612500 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:50:35.612481    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b799ddcc83fc373e734d61665661ca522f32ca5440cd742fab7395802a93d255\": container with ID starting with b799ddcc83fc373e734d61665661ca522f32ca5440cd742fab7395802a93d255 not found: ID does not exist" containerID="b799ddcc83fc373e734d61665661ca522f32ca5440cd742fab7395802a93d255"
Jul 15 20:50:35.612570 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.612511    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b799ddcc83fc373e734d61665661ca522f32ca5440cd742fab7395802a93d255"} err="failed to get container status \"b799ddcc83fc373e734d61665661ca522f32ca5440cd742fab7395802a93d255\": rpc error: code = NotFound desc = could not find container \"b799ddcc83fc373e734d61665661ca522f32ca5440cd742fab7395802a93d255\": container with ID starting with b799ddcc83fc373e734d61665661ca522f32ca5440cd742fab7395802a93d255 not found: ID does not exist"
Jul 15 20:50:35.612570 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.612535    2566 scope.go:117] "RemoveContainer" containerID="96f71b369196aef28ae04776dbbb4c281b3affffb3a304d926bfdeba9329621c"
Jul 15 20:50:35.612745 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:50:35.612731    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96f71b369196aef28ae04776dbbb4c281b3affffb3a304d926bfdeba9329621c\": container with ID starting with 96f71b369196aef28ae04776dbbb4c281b3affffb3a304d926bfdeba9329621c not found: ID does not exist" containerID="96f71b369196aef28ae04776dbbb4c281b3affffb3a304d926bfdeba9329621c"
Jul 15 20:50:35.612795 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.612747    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96f71b369196aef28ae04776dbbb4c281b3affffb3a304d926bfdeba9329621c"} err="failed to get container status \"96f71b369196aef28ae04776dbbb4c281b3affffb3a304d926bfdeba9329621c\": rpc error: code = NotFound desc = could not find container \"96f71b369196aef28ae04776dbbb4c281b3affffb3a304d926bfdeba9329621c\": container with ID starting with 96f71b369196aef28ae04776dbbb4c281b3affffb3a304d926bfdeba9329621c not found: ID does not exist"
Jul 15 20:50:35.612795 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.612759    2566 scope.go:117] "RemoveContainer" containerID="6c426b6ad12dbcd77f7c71c2742451f837b4ec1ef415d58b2cedd652cb172beb"
Jul 15 20:50:35.612990 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:50:35.612962    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c426b6ad12dbcd77f7c71c2742451f837b4ec1ef415d58b2cedd652cb172beb\": container with ID starting with 6c426b6ad12dbcd77f7c71c2742451f837b4ec1ef415d58b2cedd652cb172beb not found: ID does not exist" containerID="6c426b6ad12dbcd77f7c71c2742451f837b4ec1ef415d58b2cedd652cb172beb"
Jul 15 20:50:35.613053 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.612988    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c426b6ad12dbcd77f7c71c2742451f837b4ec1ef415d58b2cedd652cb172beb"} err="failed to get container status \"6c426b6ad12dbcd77f7c71c2742451f837b4ec1ef415d58b2cedd652cb172beb\": rpc error: code = NotFound desc = could not find container \"6c426b6ad12dbcd77f7c71c2742451f837b4ec1ef415d58b2cedd652cb172beb\": container with ID starting with 6c426b6ad12dbcd77f7c71c2742451f837b4ec1ef415d58b2cedd652cb172beb not found: ID does not exist"
Jul 15 20:50:35.613053 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.613009    2566 scope.go:117] "RemoveContainer" containerID="c0ebfdc78c0faab0b9019b484d760694c723fd61f20348fa38b76443158c6ade"
Jul 15 20:50:35.613261 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:50:35.613243    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0ebfdc78c0faab0b9019b484d760694c723fd61f20348fa38b76443158c6ade\": container with ID starting with c0ebfdc78c0faab0b9019b484d760694c723fd61f20348fa38b76443158c6ade not found: ID does not exist" containerID="c0ebfdc78c0faab0b9019b484d760694c723fd61f20348fa38b76443158c6ade"
Jul 15 20:50:35.613324 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.613264    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0ebfdc78c0faab0b9019b484d760694c723fd61f20348fa38b76443158c6ade"} err="failed to get container status \"c0ebfdc78c0faab0b9019b484d760694c723fd61f20348fa38b76443158c6ade\": rpc error: code = NotFound desc = could not find container \"c0ebfdc78c0faab0b9019b484d760694c723fd61f20348fa38b76443158c6ade\": container with ID starting with c0ebfdc78c0faab0b9019b484d760694c723fd61f20348fa38b76443158c6ade not found: ID does not exist"
Jul 15 20:50:35.613324 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.613280    2566 scope.go:117] "RemoveContainer" containerID="65dad387b1d38b98aad7bffbc9fab2c6014a9ccde5afaf16fa25a277ac084064"
Jul 15 20:50:35.613505 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:50:35.613491    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65dad387b1d38b98aad7bffbc9fab2c6014a9ccde5afaf16fa25a277ac084064\": container with ID starting with 65dad387b1d38b98aad7bffbc9fab2c6014a9ccde5afaf16fa25a277ac084064 not found: ID does not exist" containerID="65dad387b1d38b98aad7bffbc9fab2c6014a9ccde5afaf16fa25a277ac084064"
Jul 15 20:50:35.613560 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.613508    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65dad387b1d38b98aad7bffbc9fab2c6014a9ccde5afaf16fa25a277ac084064"} err="failed to get container status \"65dad387b1d38b98aad7bffbc9fab2c6014a9ccde5afaf16fa25a277ac084064\": rpc error: code = NotFound desc = could not find container \"65dad387b1d38b98aad7bffbc9fab2c6014a9ccde5afaf16fa25a277ac084064\": container with ID starting with 65dad387b1d38b98aad7bffbc9fab2c6014a9ccde5afaf16fa25a277ac084064 not found: ID does not exist"
Jul 15 20:50:35.613560 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.613520    2566 scope.go:117] "RemoveContainer" containerID="5e13a4a9f233db12707d55073929ff3354e6fbc2c0ea94a0f6a52e19c7c8fa32"
Jul 15 20:50:35.613738 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.613715    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e13a4a9f233db12707d55073929ff3354e6fbc2c0ea94a0f6a52e19c7c8fa32"} err="failed to get container status \"5e13a4a9f233db12707d55073929ff3354e6fbc2c0ea94a0f6a52e19c7c8fa32\": rpc error: code = NotFound desc = could not find container \"5e13a4a9f233db12707d55073929ff3354e6fbc2c0ea94a0f6a52e19c7c8fa32\": container with ID starting with 5e13a4a9f233db12707d55073929ff3354e6fbc2c0ea94a0f6a52e19c7c8fa32 not found: ID does not exist"
Jul 15 20:50:35.613776 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.613741    2566 scope.go:117] "RemoveContainer" containerID="85fe3276d3cc66b1ce64bbf04e29349355633893d8c54243e7471cb8b49b30dc"
Jul 15 20:50:35.614021 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.613995    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85fe3276d3cc66b1ce64bbf04e29349355633893d8c54243e7471cb8b49b30dc"} err="failed to get container status \"85fe3276d3cc66b1ce64bbf04e29349355633893d8c54243e7471cb8b49b30dc\": rpc error: code = NotFound desc = could not find container \"85fe3276d3cc66b1ce64bbf04e29349355633893d8c54243e7471cb8b49b30dc\": container with ID starting with 85fe3276d3cc66b1ce64bbf04e29349355633893d8c54243e7471cb8b49b30dc not found: ID does not exist"
Jul 15 20:50:35.614021 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.614021    2566 scope.go:117] "RemoveContainer" containerID="b799ddcc83fc373e734d61665661ca522f32ca5440cd742fab7395802a93d255"
Jul 15 20:50:35.614301 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.614280    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b799ddcc83fc373e734d61665661ca522f32ca5440cd742fab7395802a93d255"} err="failed to get container status \"b799ddcc83fc373e734d61665661ca522f32ca5440cd742fab7395802a93d255\": rpc error: code = NotFound desc = could not find container \"b799ddcc83fc373e734d61665661ca522f32ca5440cd742fab7395802a93d255\": container with ID starting with b799ddcc83fc373e734d61665661ca522f32ca5440cd742fab7395802a93d255 not found: ID does not exist"
Jul 15 20:50:35.614301 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.614301    2566 scope.go:117] "RemoveContainer" containerID="96f71b369196aef28ae04776dbbb4c281b3affffb3a304d926bfdeba9329621c"
Jul 15 20:50:35.614545 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.614523    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96f71b369196aef28ae04776dbbb4c281b3affffb3a304d926bfdeba9329621c"} err="failed to get container status \"96f71b369196aef28ae04776dbbb4c281b3affffb3a304d926bfdeba9329621c\": rpc error: code = NotFound desc = could not find container \"96f71b369196aef28ae04776dbbb4c281b3affffb3a304d926bfdeba9329621c\": container with ID starting with 96f71b369196aef28ae04776dbbb4c281b3affffb3a304d926bfdeba9329621c not found: ID does not exist"
Jul 15 20:50:35.614584 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.614557    2566 scope.go:117] "RemoveContainer" containerID="6c426b6ad12dbcd77f7c71c2742451f837b4ec1ef415d58b2cedd652cb172beb"
Jul 15 20:50:35.614818 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.614801    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c426b6ad12dbcd77f7c71c2742451f837b4ec1ef415d58b2cedd652cb172beb"} err="failed to get container status \"6c426b6ad12dbcd77f7c71c2742451f837b4ec1ef415d58b2cedd652cb172beb\": rpc error: code = NotFound desc = could not find container \"6c426b6ad12dbcd77f7c71c2742451f837b4ec1ef415d58b2cedd652cb172beb\": container with ID starting with 6c426b6ad12dbcd77f7c71c2742451f837b4ec1ef415d58b2cedd652cb172beb not found: ID does not exist"
Jul 15 20:50:35.614860 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.614818    2566 scope.go:117] "RemoveContainer" containerID="c0ebfdc78c0faab0b9019b484d760694c723fd61f20348fa38b76443158c6ade"
Jul 15 20:50:35.615119 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.615100    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0ebfdc78c0faab0b9019b484d760694c723fd61f20348fa38b76443158c6ade"} err="failed to get container status \"c0ebfdc78c0faab0b9019b484d760694c723fd61f20348fa38b76443158c6ade\": rpc error: code = NotFound desc = could not find container \"c0ebfdc78c0faab0b9019b484d760694c723fd61f20348fa38b76443158c6ade\": container with ID starting with c0ebfdc78c0faab0b9019b484d760694c723fd61f20348fa38b76443158c6ade not found: ID does not exist"
Jul 15 20:50:35.615196 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.615120    2566 scope.go:117] "RemoveContainer" containerID="65dad387b1d38b98aad7bffbc9fab2c6014a9ccde5afaf16fa25a277ac084064"
Jul 15 20:50:35.615366 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.615348    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65dad387b1d38b98aad7bffbc9fab2c6014a9ccde5afaf16fa25a277ac084064"} err="failed to get container status \"65dad387b1d38b98aad7bffbc9fab2c6014a9ccde5afaf16fa25a277ac084064\": rpc error: code = NotFound desc = could not find container \"65dad387b1d38b98aad7bffbc9fab2c6014a9ccde5afaf16fa25a277ac084064\": container with ID starting with 65dad387b1d38b98aad7bffbc9fab2c6014a9ccde5afaf16fa25a277ac084064 not found: ID does not exist"
Jul 15 20:50:35.615412 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.615368    2566 scope.go:117] "RemoveContainer" containerID="5e13a4a9f233db12707d55073929ff3354e6fbc2c0ea94a0f6a52e19c7c8fa32"
Jul 15 20:50:35.615584 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.615565    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e13a4a9f233db12707d55073929ff3354e6fbc2c0ea94a0f6a52e19c7c8fa32"} err="failed to get container status \"5e13a4a9f233db12707d55073929ff3354e6fbc2c0ea94a0f6a52e19c7c8fa32\": rpc error: code = NotFound desc = could not find container \"5e13a4a9f233db12707d55073929ff3354e6fbc2c0ea94a0f6a52e19c7c8fa32\": container with ID starting with 5e13a4a9f233db12707d55073929ff3354e6fbc2c0ea94a0f6a52e19c7c8fa32 not found: ID does not exist"
Jul 15 20:50:35.615626 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.615587    2566 scope.go:117] "RemoveContainer" containerID="85fe3276d3cc66b1ce64bbf04e29349355633893d8c54243e7471cb8b49b30dc"
Jul 15 20:50:35.615829 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.615807    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85fe3276d3cc66b1ce64bbf04e29349355633893d8c54243e7471cb8b49b30dc"} err="failed to get container status \"85fe3276d3cc66b1ce64bbf04e29349355633893d8c54243e7471cb8b49b30dc\": rpc error: code = NotFound desc = could not find container \"85fe3276d3cc66b1ce64bbf04e29349355633893d8c54243e7471cb8b49b30dc\": container with ID starting with 85fe3276d3cc66b1ce64bbf04e29349355633893d8c54243e7471cb8b49b30dc not found: ID does not exist"
Jul 15 20:50:35.615923 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.615832    2566 scope.go:117] "RemoveContainer" containerID="b799ddcc83fc373e734d61665661ca522f32ca5440cd742fab7395802a93d255"
Jul 15 20:50:35.616076 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.616049    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b799ddcc83fc373e734d61665661ca522f32ca5440cd742fab7395802a93d255"} err="failed to get container status \"b799ddcc83fc373e734d61665661ca522f32ca5440cd742fab7395802a93d255\": rpc error: code = NotFound desc = could not find container \"b799ddcc83fc373e734d61665661ca522f32ca5440cd742fab7395802a93d255\": container with ID starting with b799ddcc83fc373e734d61665661ca522f32ca5440cd742fab7395802a93d255 not found: ID does not exist"
Jul 15 20:50:35.616076 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.616072    2566 scope.go:117] "RemoveContainer" containerID="96f71b369196aef28ae04776dbbb4c281b3affffb3a304d926bfdeba9329621c"
Jul 15 20:50:35.616347 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.616324    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96f71b369196aef28ae04776dbbb4c281b3affffb3a304d926bfdeba9329621c"} err="failed to get container status \"96f71b369196aef28ae04776dbbb4c281b3affffb3a304d926bfdeba9329621c\": rpc error: code = NotFound desc = could not find container \"96f71b369196aef28ae04776dbbb4c281b3affffb3a304d926bfdeba9329621c\": container with ID starting with 96f71b369196aef28ae04776dbbb4c281b3affffb3a304d926bfdeba9329621c not found: ID does not exist"
Jul 15 20:50:35.616423 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.616348    2566 scope.go:117] "RemoveContainer" containerID="6c426b6ad12dbcd77f7c71c2742451f837b4ec1ef415d58b2cedd652cb172beb"
Jul 15 20:50:35.616686 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.616662    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c426b6ad12dbcd77f7c71c2742451f837b4ec1ef415d58b2cedd652cb172beb"} err="failed to get container status \"6c426b6ad12dbcd77f7c71c2742451f837b4ec1ef415d58b2cedd652cb172beb\": rpc error: code = NotFound desc = could not find container \"6c426b6ad12dbcd77f7c71c2742451f837b4ec1ef415d58b2cedd652cb172beb\": container with ID starting with 6c426b6ad12dbcd77f7c71c2742451f837b4ec1ef415d58b2cedd652cb172beb not found: ID does not exist"
Jul 15 20:50:35.616742 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.616689    2566 scope.go:117] "RemoveContainer" containerID="c0ebfdc78c0faab0b9019b484d760694c723fd61f20348fa38b76443158c6ade"
Jul 15 20:50:35.616981 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.616963    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0ebfdc78c0faab0b9019b484d760694c723fd61f20348fa38b76443158c6ade"} err="failed to get container status \"c0ebfdc78c0faab0b9019b484d760694c723fd61f20348fa38b76443158c6ade\": rpc error: code = NotFound desc = could not find container \"c0ebfdc78c0faab0b9019b484d760694c723fd61f20348fa38b76443158c6ade\": container with ID starting with c0ebfdc78c0faab0b9019b484d760694c723fd61f20348fa38b76443158c6ade not found: ID does not exist"
Jul 15 20:50:35.617054 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.616982    2566 scope.go:117] "RemoveContainer" containerID="65dad387b1d38b98aad7bffbc9fab2c6014a9ccde5afaf16fa25a277ac084064"
Jul 15 20:50:35.617232 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.617211    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65dad387b1d38b98aad7bffbc9fab2c6014a9ccde5afaf16fa25a277ac084064"} err="failed to get container status \"65dad387b1d38b98aad7bffbc9fab2c6014a9ccde5afaf16fa25a277ac084064\": rpc error: code = NotFound desc = could not find container \"65dad387b1d38b98aad7bffbc9fab2c6014a9ccde5afaf16fa25a277ac084064\": container with ID starting with 65dad387b1d38b98aad7bffbc9fab2c6014a9ccde5afaf16fa25a277ac084064 not found: ID does not exist"
Jul 15 20:50:35.617302 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.617234    2566 scope.go:117] "RemoveContainer" containerID="5e13a4a9f233db12707d55073929ff3354e6fbc2c0ea94a0f6a52e19c7c8fa32"
Jul 15 20:50:35.617471 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.617456    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e13a4a9f233db12707d55073929ff3354e6fbc2c0ea94a0f6a52e19c7c8fa32"} err="failed to get container status \"5e13a4a9f233db12707d55073929ff3354e6fbc2c0ea94a0f6a52e19c7c8fa32\": rpc error: code = NotFound desc = could not find container \"5e13a4a9f233db12707d55073929ff3354e6fbc2c0ea94a0f6a52e19c7c8fa32\": container with ID starting with 5e13a4a9f233db12707d55073929ff3354e6fbc2c0ea94a0f6a52e19c7c8fa32 not found: ID does not exist"
Jul 15 20:50:35.617531 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.617472    2566 scope.go:117] "RemoveContainer" containerID="85fe3276d3cc66b1ce64bbf04e29349355633893d8c54243e7471cb8b49b30dc"
Jul 15 20:50:35.617718 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.617694    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85fe3276d3cc66b1ce64bbf04e29349355633893d8c54243e7471cb8b49b30dc"} err="failed to get container status \"85fe3276d3cc66b1ce64bbf04e29349355633893d8c54243e7471cb8b49b30dc\": rpc error: code = NotFound desc = could not find container \"85fe3276d3cc66b1ce64bbf04e29349355633893d8c54243e7471cb8b49b30dc\": container with ID starting with 85fe3276d3cc66b1ce64bbf04e29349355633893d8c54243e7471cb8b49b30dc not found: ID does not exist"
Jul 15 20:50:35.617772 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.617723    2566 scope.go:117] "RemoveContainer" containerID="b799ddcc83fc373e734d61665661ca522f32ca5440cd742fab7395802a93d255"
Jul 15 20:50:35.617946 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.617924    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b799ddcc83fc373e734d61665661ca522f32ca5440cd742fab7395802a93d255"} err="failed to get container status \"b799ddcc83fc373e734d61665661ca522f32ca5440cd742fab7395802a93d255\": rpc error: code = NotFound desc = could not find container \"b799ddcc83fc373e734d61665661ca522f32ca5440cd742fab7395802a93d255\": container with ID starting with b799ddcc83fc373e734d61665661ca522f32ca5440cd742fab7395802a93d255 not found: ID does not exist"
Jul 15 20:50:35.617946 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.617946    2566 scope.go:117] "RemoveContainer" containerID="96f71b369196aef28ae04776dbbb4c281b3affffb3a304d926bfdeba9329621c"
Jul 15 20:50:35.618189 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.618170    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96f71b369196aef28ae04776dbbb4c281b3affffb3a304d926bfdeba9329621c"} err="failed to get container status \"96f71b369196aef28ae04776dbbb4c281b3affffb3a304d926bfdeba9329621c\": rpc error: code = NotFound desc = could not find container \"96f71b369196aef28ae04776dbbb4c281b3affffb3a304d926bfdeba9329621c\": container with ID starting with 96f71b369196aef28ae04776dbbb4c281b3affffb3a304d926bfdeba9329621c not found: ID does not exist"
Jul 15 20:50:35.618264 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.618192    2566 scope.go:117] "RemoveContainer" containerID="6c426b6ad12dbcd77f7c71c2742451f837b4ec1ef415d58b2cedd652cb172beb"
Jul 15 20:50:35.618423 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.618405    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c426b6ad12dbcd77f7c71c2742451f837b4ec1ef415d58b2cedd652cb172beb"} err="failed to get container status \"6c426b6ad12dbcd77f7c71c2742451f837b4ec1ef415d58b2cedd652cb172beb\": rpc error: code = NotFound desc = could not find container \"6c426b6ad12dbcd77f7c71c2742451f837b4ec1ef415d58b2cedd652cb172beb\": container with ID starting with 6c426b6ad12dbcd77f7c71c2742451f837b4ec1ef415d58b2cedd652cb172beb not found: ID does not exist"
Jul 15 20:50:35.618490 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.618424    2566 scope.go:117] "RemoveContainer" containerID="c0ebfdc78c0faab0b9019b484d760694c723fd61f20348fa38b76443158c6ade"
Jul 15 20:50:35.618654 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.618637    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0ebfdc78c0faab0b9019b484d760694c723fd61f20348fa38b76443158c6ade"} err="failed to get container status \"c0ebfdc78c0faab0b9019b484d760694c723fd61f20348fa38b76443158c6ade\": rpc error: code = NotFound desc = could not find container \"c0ebfdc78c0faab0b9019b484d760694c723fd61f20348fa38b76443158c6ade\": container with ID starting with c0ebfdc78c0faab0b9019b484d760694c723fd61f20348fa38b76443158c6ade not found: ID does not exist"
Jul 15 20:50:35.618654 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.618654    2566 scope.go:117] "RemoveContainer" containerID="65dad387b1d38b98aad7bffbc9fab2c6014a9ccde5afaf16fa25a277ac084064"
Jul 15 20:50:35.618940 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.618917    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65dad387b1d38b98aad7bffbc9fab2c6014a9ccde5afaf16fa25a277ac084064"} err="failed to get container status \"65dad387b1d38b98aad7bffbc9fab2c6014a9ccde5afaf16fa25a277ac084064\": rpc error: code = NotFound desc = could not find container \"65dad387b1d38b98aad7bffbc9fab2c6014a9ccde5afaf16fa25a277ac084064\": container with ID starting with 65dad387b1d38b98aad7bffbc9fab2c6014a9ccde5afaf16fa25a277ac084064 not found: ID does not exist"
Jul 15 20:50:35.618940 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.618940    2566 scope.go:117] "RemoveContainer" containerID="5e13a4a9f233db12707d55073929ff3354e6fbc2c0ea94a0f6a52e19c7c8fa32"
Jul 15 20:50:35.619129 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.619113    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e13a4a9f233db12707d55073929ff3354e6fbc2c0ea94a0f6a52e19c7c8fa32"} err="failed to get container status \"5e13a4a9f233db12707d55073929ff3354e6fbc2c0ea94a0f6a52e19c7c8fa32\": rpc error: code = NotFound desc = could not find container \"5e13a4a9f233db12707d55073929ff3354e6fbc2c0ea94a0f6a52e19c7c8fa32\": container with ID starting with 5e13a4a9f233db12707d55073929ff3354e6fbc2c0ea94a0f6a52e19c7c8fa32 not found: ID does not exist"
Jul 15 20:50:35.619129 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.619129    2566 scope.go:117] "RemoveContainer" containerID="85fe3276d3cc66b1ce64bbf04e29349355633893d8c54243e7471cb8b49b30dc"
Jul 15 20:50:35.619368 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.619353    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85fe3276d3cc66b1ce64bbf04e29349355633893d8c54243e7471cb8b49b30dc"} err="failed to get container status \"85fe3276d3cc66b1ce64bbf04e29349355633893d8c54243e7471cb8b49b30dc\": rpc error: code = NotFound desc = could not find container \"85fe3276d3cc66b1ce64bbf04e29349355633893d8c54243e7471cb8b49b30dc\": container with ID starting with 85fe3276d3cc66b1ce64bbf04e29349355633893d8c54243e7471cb8b49b30dc not found: ID does not exist"
Jul 15 20:50:35.619412 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.619368    2566 scope.go:117] "RemoveContainer" containerID="b799ddcc83fc373e734d61665661ca522f32ca5440cd742fab7395802a93d255"
Jul 15 20:50:35.619570 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.619550    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b799ddcc83fc373e734d61665661ca522f32ca5440cd742fab7395802a93d255"} err="failed to get container status \"b799ddcc83fc373e734d61665661ca522f32ca5440cd742fab7395802a93d255\": rpc error: code = NotFound desc = could not find container \"b799ddcc83fc373e734d61665661ca522f32ca5440cd742fab7395802a93d255\": container with ID starting with b799ddcc83fc373e734d61665661ca522f32ca5440cd742fab7395802a93d255 not found: ID does not exist"
Jul 15 20:50:35.619607 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.619571    2566 scope.go:117] "RemoveContainer" containerID="96f71b369196aef28ae04776dbbb4c281b3affffb3a304d926bfdeba9329621c"
Jul 15 20:50:35.619783 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.619767    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96f71b369196aef28ae04776dbbb4c281b3affffb3a304d926bfdeba9329621c"} err="failed to get container status \"96f71b369196aef28ae04776dbbb4c281b3affffb3a304d926bfdeba9329621c\": rpc error: code = NotFound desc = could not find container \"96f71b369196aef28ae04776dbbb4c281b3affffb3a304d926bfdeba9329621c\": container with ID starting with 96f71b369196aef28ae04776dbbb4c281b3affffb3a304d926bfdeba9329621c not found: ID does not exist"
Jul 15 20:50:35.619846 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.619783    2566 scope.go:117] "RemoveContainer" containerID="6c426b6ad12dbcd77f7c71c2742451f837b4ec1ef415d58b2cedd652cb172beb"
Jul 15 20:50:35.620046 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.620022    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c426b6ad12dbcd77f7c71c2742451f837b4ec1ef415d58b2cedd652cb172beb"} err="failed to get container status \"6c426b6ad12dbcd77f7c71c2742451f837b4ec1ef415d58b2cedd652cb172beb\": rpc error: code = NotFound desc = could not find container \"6c426b6ad12dbcd77f7c71c2742451f837b4ec1ef415d58b2cedd652cb172beb\": container with ID starting with 6c426b6ad12dbcd77f7c71c2742451f837b4ec1ef415d58b2cedd652cb172beb not found: ID does not exist"
Jul 15 20:50:35.620046 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.620041    2566 scope.go:117] "RemoveContainer" containerID="c0ebfdc78c0faab0b9019b484d760694c723fd61f20348fa38b76443158c6ade"
Jul 15 20:50:35.620281 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.620265    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0ebfdc78c0faab0b9019b484d760694c723fd61f20348fa38b76443158c6ade"} err="failed to get container status \"c0ebfdc78c0faab0b9019b484d760694c723fd61f20348fa38b76443158c6ade\": rpc error: code = NotFound desc = could not find container \"c0ebfdc78c0faab0b9019b484d760694c723fd61f20348fa38b76443158c6ade\": container with ID starting with c0ebfdc78c0faab0b9019b484d760694c723fd61f20348fa38b76443158c6ade not found: ID does not exist"
Jul 15 20:50:35.620281 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.620281    2566 scope.go:117] "RemoveContainer" containerID="65dad387b1d38b98aad7bffbc9fab2c6014a9ccde5afaf16fa25a277ac084064"
Jul 15 20:50:35.620495 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.620476    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65dad387b1d38b98aad7bffbc9fab2c6014a9ccde5afaf16fa25a277ac084064"} err="failed to get container status \"65dad387b1d38b98aad7bffbc9fab2c6014a9ccde5afaf16fa25a277ac084064\": rpc error: code = NotFound desc = could not find container \"65dad387b1d38b98aad7bffbc9fab2c6014a9ccde5afaf16fa25a277ac084064\": container with ID starting with 65dad387b1d38b98aad7bffbc9fab2c6014a9ccde5afaf16fa25a277ac084064 not found: ID does not exist"
Jul 15 20:50:35.620495 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.620494    2566 scope.go:117] "RemoveContainer" containerID="5e13a4a9f233db12707d55073929ff3354e6fbc2c0ea94a0f6a52e19c7c8fa32"
Jul 15 20:50:35.620724 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.620700    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e13a4a9f233db12707d55073929ff3354e6fbc2c0ea94a0f6a52e19c7c8fa32"} err="failed to get container status \"5e13a4a9f233db12707d55073929ff3354e6fbc2c0ea94a0f6a52e19c7c8fa32\": rpc error: code = NotFound desc = could not find container \"5e13a4a9f233db12707d55073929ff3354e6fbc2c0ea94a0f6a52e19c7c8fa32\": container with ID starting with 5e13a4a9f233db12707d55073929ff3354e6fbc2c0ea94a0f6a52e19c7c8fa32 not found: ID does not exist"
Jul 15 20:50:35.620724 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.620723    2566 scope.go:117] "RemoveContainer" containerID="85fe3276d3cc66b1ce64bbf04e29349355633893d8c54243e7471cb8b49b30dc"
Jul 15 20:50:35.620969 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.620952    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85fe3276d3cc66b1ce64bbf04e29349355633893d8c54243e7471cb8b49b30dc"} err="failed to get container status \"85fe3276d3cc66b1ce64bbf04e29349355633893d8c54243e7471cb8b49b30dc\": rpc error: code = NotFound desc = could not find container \"85fe3276d3cc66b1ce64bbf04e29349355633893d8c54243e7471cb8b49b30dc\": container with ID starting with 85fe3276d3cc66b1ce64bbf04e29349355633893d8c54243e7471cb8b49b30dc not found: ID does not exist"
Jul 15 20:50:35.621021 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.620970    2566 scope.go:117] "RemoveContainer" containerID="b799ddcc83fc373e734d61665661ca522f32ca5440cd742fab7395802a93d255"
Jul 15 20:50:35.621171 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.621155    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b799ddcc83fc373e734d61665661ca522f32ca5440cd742fab7395802a93d255"} err="failed to get container status \"b799ddcc83fc373e734d61665661ca522f32ca5440cd742fab7395802a93d255\": rpc error: code = NotFound desc = could not find container \"b799ddcc83fc373e734d61665661ca522f32ca5440cd742fab7395802a93d255\": container with ID starting with b799ddcc83fc373e734d61665661ca522f32ca5440cd742fab7395802a93d255 not found: ID does not exist"
Jul 15 20:50:35.621221 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.621172    2566 scope.go:117] "RemoveContainer" containerID="96f71b369196aef28ae04776dbbb4c281b3affffb3a304d926bfdeba9329621c"
Jul 15 20:50:35.621408 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.621390    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96f71b369196aef28ae04776dbbb4c281b3affffb3a304d926bfdeba9329621c"} err="failed to get container status \"96f71b369196aef28ae04776dbbb4c281b3affffb3a304d926bfdeba9329621c\": rpc error: code = NotFound desc = could not find container \"96f71b369196aef28ae04776dbbb4c281b3affffb3a304d926bfdeba9329621c\": container with ID starting with 96f71b369196aef28ae04776dbbb4c281b3affffb3a304d926bfdeba9329621c not found: ID does not exist"
Jul 15 20:50:35.621476 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.621408    2566 scope.go:117] "RemoveContainer" containerID="6c426b6ad12dbcd77f7c71c2742451f837b4ec1ef415d58b2cedd652cb172beb"
Jul 15 20:50:35.621626 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.621607    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c426b6ad12dbcd77f7c71c2742451f837b4ec1ef415d58b2cedd652cb172beb"} err="failed to get container status \"6c426b6ad12dbcd77f7c71c2742451f837b4ec1ef415d58b2cedd652cb172beb\": rpc error: code = NotFound desc = could not find container \"6c426b6ad12dbcd77f7c71c2742451f837b4ec1ef415d58b2cedd652cb172beb\": container with ID starting with 6c426b6ad12dbcd77f7c71c2742451f837b4ec1ef415d58b2cedd652cb172beb not found: ID does not exist"
Jul 15 20:50:35.621626 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.621625    2566 scope.go:117] "RemoveContainer" containerID="c0ebfdc78c0faab0b9019b484d760694c723fd61f20348fa38b76443158c6ade"
Jul 15 20:50:35.621861 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.621840    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0ebfdc78c0faab0b9019b484d760694c723fd61f20348fa38b76443158c6ade"} err="failed to get container status \"c0ebfdc78c0faab0b9019b484d760694c723fd61f20348fa38b76443158c6ade\": rpc error: code = NotFound desc = could not find container \"c0ebfdc78c0faab0b9019b484d760694c723fd61f20348fa38b76443158c6ade\": container with ID starting with c0ebfdc78c0faab0b9019b484d760694c723fd61f20348fa38b76443158c6ade not found: ID does not exist"
Jul 15 20:50:35.621934 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.621862    2566 scope.go:117] "RemoveContainer" containerID="65dad387b1d38b98aad7bffbc9fab2c6014a9ccde5afaf16fa25a277ac084064"
Jul 15 20:50:35.622129 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.622108    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65dad387b1d38b98aad7bffbc9fab2c6014a9ccde5afaf16fa25a277ac084064"} err="failed to get container status \"65dad387b1d38b98aad7bffbc9fab2c6014a9ccde5afaf16fa25a277ac084064\": rpc error: code = NotFound desc = could not find container \"65dad387b1d38b98aad7bffbc9fab2c6014a9ccde5afaf16fa25a277ac084064\": container with ID starting with 65dad387b1d38b98aad7bffbc9fab2c6014a9ccde5afaf16fa25a277ac084064 not found: ID does not exist"
Jul 15 20:50:35.705601 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.705566    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-thanos-prometheus-http-client-file\") pod \"aaa43101-957c-4064-92db-ba62aabdd8d9\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") "
Jul 15 20:50:35.705758 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.705616    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aaa43101-957c-4064-92db-ba62aabdd8d9-config-out\") pod \"aaa43101-957c-4064-92db-ba62aabdd8d9\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") "
Jul 15 20:50:35.705758 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.705643    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-secret-prometheus-k8s-tls\") pod \"aaa43101-957c-4064-92db-ba62aabdd8d9\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") "
Jul 15 20:50:35.705758 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.705667    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-config\") pod \"aaa43101-957c-4064-92db-ba62aabdd8d9\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") "
Jul 15 20:50:35.705758 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.705690    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-secret-grpc-tls\") pod \"aaa43101-957c-4064-92db-ba62aabdd8d9\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") "
Jul 15 20:50:35.705758 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.705707    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aaa43101-957c-4064-92db-ba62aabdd8d9-configmap-kubelet-serving-ca-bundle\") pod \"aaa43101-957c-4064-92db-ba62aabdd8d9\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") "
Jul 15 20:50:35.705758 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.705734    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/aaa43101-957c-4064-92db-ba62aabdd8d9-prometheus-k8s-rulefiles-1\") pod \"aaa43101-957c-4064-92db-ba62aabdd8d9\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") "
Jul 15 20:50:35.705758 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.705758    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/aaa43101-957c-4064-92db-ba62aabdd8d9-prometheus-k8s-rulefiles-0\") pod \"aaa43101-957c-4064-92db-ba62aabdd8d9\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") "
Jul 15 20:50:35.706156 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.705782    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2b4j\" (UniqueName: \"kubernetes.io/projected/aaa43101-957c-4064-92db-ba62aabdd8d9-kube-api-access-s2b4j\") pod \"aaa43101-957c-4064-92db-ba62aabdd8d9\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") "
Jul 15 20:50:35.706156 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.706128    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaa43101-957c-4064-92db-ba62aabdd8d9-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "aaa43101-957c-4064-92db-ba62aabdd8d9" (UID: "aaa43101-957c-4064-92db-ba62aabdd8d9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 20:50:35.706454 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.706420    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaa43101-957c-4064-92db-ba62aabdd8d9-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "aaa43101-957c-4064-92db-ba62aabdd8d9" (UID: "aaa43101-957c-4064-92db-ba62aabdd8d9"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 20:50:35.706534 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.706486    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aaa43101-957c-4064-92db-ba62aabdd8d9-configmap-serving-certs-ca-bundle\") pod \"aaa43101-957c-4064-92db-ba62aabdd8d9\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") "
Jul 15 20:50:35.706534 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.706527    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-secret-metrics-client-certs\") pod \"aaa43101-957c-4064-92db-ba62aabdd8d9\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") "
Jul 15 20:50:35.706642 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.706558    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"aaa43101-957c-4064-92db-ba62aabdd8d9\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") "
Jul 15 20:50:35.706642 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.706622    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-secret-kube-rbac-proxy\") pod \"aaa43101-957c-4064-92db-ba62aabdd8d9\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") "
Jul 15 20:50:35.706741 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.706652    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"aaa43101-957c-4064-92db-ba62aabdd8d9\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") "
Jul 15 20:50:35.706741 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.706676    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/aaa43101-957c-4064-92db-ba62aabdd8d9-prometheus-k8s-db\") pod \"aaa43101-957c-4064-92db-ba62aabdd8d9\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") "
Jul 15 20:50:35.706741 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.706705    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/aaa43101-957c-4064-92db-ba62aabdd8d9-prometheus-k8s-rulefiles-2\") pod \"aaa43101-957c-4064-92db-ba62aabdd8d9\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") "
Jul 15 20:50:35.706741 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.706735    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-web-config\") pod \"aaa43101-957c-4064-92db-ba62aabdd8d9\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") "
Jul 15 20:50:35.708246 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.706761    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aaa43101-957c-4064-92db-ba62aabdd8d9-tls-assets\") pod \"aaa43101-957c-4064-92db-ba62aabdd8d9\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") "
Jul 15 20:50:35.708246 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.706868    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aaa43101-957c-4064-92db-ba62aabdd8d9-prometheus-trusted-ca-bundle\") pod \"aaa43101-957c-4064-92db-ba62aabdd8d9\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") "
Jul 15 20:50:35.708246 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.706940    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aaa43101-957c-4064-92db-ba62aabdd8d9-configmap-metrics-client-ca\") pod \"aaa43101-957c-4064-92db-ba62aabdd8d9\" (UID: \"aaa43101-957c-4064-92db-ba62aabdd8d9\") "
Jul 15 20:50:35.708246 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.707321    2566 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aaa43101-957c-4064-92db-ba62aabdd8d9-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 20:50:35.708246 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.707344    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/aaa43101-957c-4064-92db-ba62aabdd8d9-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 20:50:35.708246 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.707577    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaa43101-957c-4064-92db-ba62aabdd8d9-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "aaa43101-957c-4064-92db-ba62aabdd8d9" (UID: "aaa43101-957c-4064-92db-ba62aabdd8d9"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 20:50:35.708246 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.707689    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaa43101-957c-4064-92db-ba62aabdd8d9-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "aaa43101-957c-4064-92db-ba62aabdd8d9" (UID: "aaa43101-957c-4064-92db-ba62aabdd8d9"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 20:50:35.708246 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.708177    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaa43101-957c-4064-92db-ba62aabdd8d9-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "aaa43101-957c-4064-92db-ba62aabdd8d9" (UID: "aaa43101-957c-4064-92db-ba62aabdd8d9"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 20:50:35.708658 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.708336    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaa43101-957c-4064-92db-ba62aabdd8d9-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "aaa43101-957c-4064-92db-ba62aabdd8d9" (UID: "aaa43101-957c-4064-92db-ba62aabdd8d9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 20:50:35.708658 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.708430    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaa43101-957c-4064-92db-ba62aabdd8d9-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "aaa43101-957c-4064-92db-ba62aabdd8d9" (UID: "aaa43101-957c-4064-92db-ba62aabdd8d9"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 20:50:35.708764 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.708711    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "aaa43101-957c-4064-92db-ba62aabdd8d9" (UID: "aaa43101-957c-4064-92db-ba62aabdd8d9"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 20:50:35.708819 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.708792    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-config" (OuterVolumeSpecName: "config") pod "aaa43101-957c-4064-92db-ba62aabdd8d9" (UID: "aaa43101-957c-4064-92db-ba62aabdd8d9"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 20:50:35.709143 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.709109    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaa43101-957c-4064-92db-ba62aabdd8d9-config-out" (OuterVolumeSpecName: "config-out") pod "aaa43101-957c-4064-92db-ba62aabdd8d9" (UID: "aaa43101-957c-4064-92db-ba62aabdd8d9"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 20:50:35.709248 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.709225    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaa43101-957c-4064-92db-ba62aabdd8d9-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "aaa43101-957c-4064-92db-ba62aabdd8d9" (UID: "aaa43101-957c-4064-92db-ba62aabdd8d9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 20:50:35.709799 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.709771    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "aaa43101-957c-4064-92db-ba62aabdd8d9" (UID: "aaa43101-957c-4064-92db-ba62aabdd8d9"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 20:50:35.710229 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.710097    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaa43101-957c-4064-92db-ba62aabdd8d9-kube-api-access-s2b4j" (OuterVolumeSpecName: "kube-api-access-s2b4j") pod "aaa43101-957c-4064-92db-ba62aabdd8d9" (UID: "aaa43101-957c-4064-92db-ba62aabdd8d9"). InnerVolumeSpecName "kube-api-access-s2b4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 20:50:35.710229 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.710200    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "aaa43101-957c-4064-92db-ba62aabdd8d9" (UID: "aaa43101-957c-4064-92db-ba62aabdd8d9"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 20:50:35.710401 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.710350    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "aaa43101-957c-4064-92db-ba62aabdd8d9" (UID: "aaa43101-957c-4064-92db-ba62aabdd8d9"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 20:50:35.710401 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.710376    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "aaa43101-957c-4064-92db-ba62aabdd8d9" (UID: "aaa43101-957c-4064-92db-ba62aabdd8d9"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 20:50:35.710593 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.710574    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "aaa43101-957c-4064-92db-ba62aabdd8d9" (UID: "aaa43101-957c-4064-92db-ba62aabdd8d9"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 20:50:35.710822 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.710797    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaa43101-957c-4064-92db-ba62aabdd8d9-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "aaa43101-957c-4064-92db-ba62aabdd8d9" (UID: "aaa43101-957c-4064-92db-ba62aabdd8d9"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 20:50:35.711230 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.711204    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "aaa43101-957c-4064-92db-ba62aabdd8d9" (UID: "aaa43101-957c-4064-92db-ba62aabdd8d9"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 20:50:35.720558 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.720537    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-web-config" (OuterVolumeSpecName: "web-config") pod "aaa43101-957c-4064-92db-ba62aabdd8d9" (UID: "aaa43101-957c-4064-92db-ba62aabdd8d9"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 20:50:35.807994 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.807956    2566 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aaa43101-957c-4064-92db-ba62aabdd8d9-configmap-metrics-client-ca\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 20:50:35.807994 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.807988    2566 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 20:50:35.807994 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.807999    2566 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aaa43101-957c-4064-92db-ba62aabdd8d9-config-out\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 20:50:35.808210 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.808009    2566 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-secret-prometheus-k8s-tls\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 20:50:35.808210 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.808018    2566 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-config\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 20:50:35.808210 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.808027    2566 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-secret-grpc-tls\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 20:50:35.808210 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.808035    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/aaa43101-957c-4064-92db-ba62aabdd8d9-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 20:50:35.808210 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.808043    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2b4j\" (UniqueName: \"kubernetes.io/projected/aaa43101-957c-4064-92db-ba62aabdd8d9-kube-api-access-s2b4j\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 20:50:35.808210 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.808051    2566 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aaa43101-957c-4064-92db-ba62aabdd8d9-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 20:50:35.808210 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.808059    2566 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-secret-metrics-client-certs\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 20:50:35.808210 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.808068    2566 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 20:50:35.808210 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.808078    2566 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-secret-kube-rbac-proxy\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 20:50:35.808210 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.808086    2566 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 20:50:35.808210 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.808094    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/aaa43101-957c-4064-92db-ba62aabdd8d9-prometheus-k8s-db\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 20:50:35.808210 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.808102    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/aaa43101-957c-4064-92db-ba62aabdd8d9-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 20:50:35.808210 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.808110    2566 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aaa43101-957c-4064-92db-ba62aabdd8d9-web-config\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 20:50:35.808210 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.808118    2566 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aaa43101-957c-4064-92db-ba62aabdd8d9-tls-assets\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 20:50:35.808210 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.808125    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aaa43101-957c-4064-92db-ba62aabdd8d9-prometheus-trusted-ca-bundle\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 20:50:35.884881 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.884809    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 15 20:50:35.889612 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.889588    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 15 20:50:35.923245 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.923215    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 15 20:50:35.923479 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.923466    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaa43101-957c-4064-92db-ba62aabdd8d9" containerName="kube-rbac-proxy-thanos"
Jul 15 20:50:35.923520 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.923481    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaa43101-957c-4064-92db-ba62aabdd8d9" containerName="kube-rbac-proxy-thanos"
Jul 15 20:50:35.923520 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.923491    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaa43101-957c-4064-92db-ba62aabdd8d9" containerName="prometheus"
Jul 15 20:50:35.923520 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.923497    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaa43101-957c-4064-92db-ba62aabdd8d9" containerName="prometheus"
Jul 15 20:50:35.923520 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.923506    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaa43101-957c-4064-92db-ba62aabdd8d9" containerName="config-reloader"
Jul 15 20:50:35.923520 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.923511    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaa43101-957c-4064-92db-ba62aabdd8d9" containerName="config-reloader"
Jul 15 20:50:35.923520 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.923517    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaa43101-957c-4064-92db-ba62aabdd8d9" containerName="kube-rbac-proxy-web"
Jul 15 20:50:35.923520 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.923522    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaa43101-957c-4064-92db-ba62aabdd8d9" containerName="kube-rbac-proxy-web"
Jul 15 20:50:35.923708 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.923529    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaa43101-957c-4064-92db-ba62aabdd8d9" containerName="init-config-reloader"
Jul 15 20:50:35.923708 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.923534    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaa43101-957c-4064-92db-ba62aabdd8d9" containerName="init-config-reloader"
Jul 15 20:50:35.923708 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.923543    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaa43101-957c-4064-92db-ba62aabdd8d9" containerName="thanos-sidecar"
Jul 15 20:50:35.923708 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.923548    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaa43101-957c-4064-92db-ba62aabdd8d9" containerName="thanos-sidecar"
Jul 15 20:50:35.923708 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.923554    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaa43101-957c-4064-92db-ba62aabdd8d9" containerName="kube-rbac-proxy"
Jul 15 20:50:35.923708 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.923559    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaa43101-957c-4064-92db-ba62aabdd8d9" containerName="kube-rbac-proxy"
Jul 15 20:50:35.923708 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.923609    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="aaa43101-957c-4064-92db-ba62aabdd8d9" containerName="thanos-sidecar"
Jul 15 20:50:35.923708 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.923617    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="aaa43101-957c-4064-92db-ba62aabdd8d9" containerName="prometheus"
Jul 15 20:50:35.923708 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.923622    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="aaa43101-957c-4064-92db-ba62aabdd8d9" containerName="kube-rbac-proxy"
Jul 15 20:50:35.923708 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.923628    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="aaa43101-957c-4064-92db-ba62aabdd8d9" containerName="config-reloader"
Jul 15 20:50:35.923708 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.923633    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="aaa43101-957c-4064-92db-ba62aabdd8d9" containerName="kube-rbac-proxy-web"
Jul 15 20:50:35.923708 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.923639    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="aaa43101-957c-4064-92db-ba62aabdd8d9" containerName="kube-rbac-proxy-thanos"
Jul 15 20:50:35.928172 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.928156    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:35.931394 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.931368    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7qhn7\""
Jul 15 20:50:35.931654 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.931637    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 15 20:50:35.931734 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.931638    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 15 20:50:35.931734 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.931676    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 15 20:50:35.931968 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.931950    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 15 20:50:35.932059 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.931972    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 15 20:50:35.932059 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.931986    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 15 20:50:35.932169 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.932072    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 15 20:50:35.932226 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.932165    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 15 20:50:35.932271 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.932262    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 15 20:50:35.932402 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.932388    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-471ecd56nuucl\""
Jul 15 20:50:35.932466 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.932452    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 15 20:50:35.932526 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.932458    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 15 20:50:35.932857 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.932842    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 15 20:50:35.932953 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.932915    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 15 20:50:35.936150 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.936130    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 15 20:50:35.938345 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.938330    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 15 20:50:35.945561 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:35.945541    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 15 20:50:36.010011 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.009977    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f6164d2a-4adb-4ef7-8443-d53e915eb70e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.010156 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.010036    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f6164d2a-4adb-4ef7-8443-d53e915eb70e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.010156 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.010071    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f6164d2a-4adb-4ef7-8443-d53e915eb70e-config\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.010156 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.010100    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f6164d2a-4adb-4ef7-8443-d53e915eb70e-web-config\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.010156 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.010127    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6164d2a-4adb-4ef7-8443-d53e915eb70e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.010279 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.010181    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f6164d2a-4adb-4ef7-8443-d53e915eb70e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.010279 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.010206    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f6164d2a-4adb-4ef7-8443-d53e915eb70e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.010279 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.010232    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6164d2a-4adb-4ef7-8443-d53e915eb70e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.010279 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.010258    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f6164d2a-4adb-4ef7-8443-d53e915eb70e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.010403 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.010289    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f6164d2a-4adb-4ef7-8443-d53e915eb70e-config-out\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.010403 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.010310    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f6164d2a-4adb-4ef7-8443-d53e915eb70e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.010403 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.010360    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f6164d2a-4adb-4ef7-8443-d53e915eb70e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.010403 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.010381    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f6164d2a-4adb-4ef7-8443-d53e915eb70e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.010515 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.010438    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6164d2a-4adb-4ef7-8443-d53e915eb70e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.010515 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.010471    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f6164d2a-4adb-4ef7-8443-d53e915eb70e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.010515 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.010488    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f6164d2a-4adb-4ef7-8443-d53e915eb70e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.010515 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.010503    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c5gx\" (UniqueName: \"kubernetes.io/projected/f6164d2a-4adb-4ef7-8443-d53e915eb70e-kube-api-access-6c5gx\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.010632 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.010520    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f6164d2a-4adb-4ef7-8443-d53e915eb70e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.010632 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.010540    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6164d2a-4adb-4ef7-8443-d53e915eb70e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.010632 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.010560    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f6164d2a-4adb-4ef7-8443-d53e915eb70e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.111107 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.111072    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f6164d2a-4adb-4ef7-8443-d53e915eb70e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.111107 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.111107    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f6164d2a-4adb-4ef7-8443-d53e915eb70e-config-out\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.111334 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.111126    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f6164d2a-4adb-4ef7-8443-d53e915eb70e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.111334 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.111178    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f6164d2a-4adb-4ef7-8443-d53e915eb70e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.111334 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.111201    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f6164d2a-4adb-4ef7-8443-d53e915eb70e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.111334 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.111230    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6164d2a-4adb-4ef7-8443-d53e915eb70e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.111529 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.111348    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f6164d2a-4adb-4ef7-8443-d53e915eb70e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.111529 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.111398    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f6164d2a-4adb-4ef7-8443-d53e915eb70e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.111529 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.111428    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c5gx\" (UniqueName: \"kubernetes.io/projected/f6164d2a-4adb-4ef7-8443-d53e915eb70e-kube-api-access-6c5gx\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.111529 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.111458    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f6164d2a-4adb-4ef7-8443-d53e915eb70e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.111529 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.111490    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6164d2a-4adb-4ef7-8443-d53e915eb70e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.111529 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.111516    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f6164d2a-4adb-4ef7-8443-d53e915eb70e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.111816 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.111566    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f6164d2a-4adb-4ef7-8443-d53e915eb70e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.111816 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.111608    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f6164d2a-4adb-4ef7-8443-d53e915eb70e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.111816 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.111635    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f6164d2a-4adb-4ef7-8443-d53e915eb70e-config\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.111816 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.111661    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f6164d2a-4adb-4ef7-8443-d53e915eb70e-web-config\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.111816 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.111686    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6164d2a-4adb-4ef7-8443-d53e915eb70e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.111816 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.111719    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f6164d2a-4adb-4ef7-8443-d53e915eb70e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.111816 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.111743    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f6164d2a-4adb-4ef7-8443-d53e915eb70e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.111816 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.111773    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6164d2a-4adb-4ef7-8443-d53e915eb70e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.112208 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.112169    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f6164d2a-4adb-4ef7-8443-d53e915eb70e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.112332 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.112309    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f6164d2a-4adb-4ef7-8443-d53e915eb70e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.113050 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.113017    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6164d2a-4adb-4ef7-8443-d53e915eb70e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.113329 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.113308    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6164d2a-4adb-4ef7-8443-d53e915eb70e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.114106 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.114052    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f6164d2a-4adb-4ef7-8443-d53e915eb70e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.114911 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.114519    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6164d2a-4adb-4ef7-8443-d53e915eb70e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.114911 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.114839    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6164d2a-4adb-4ef7-8443-d53e915eb70e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.115062 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.114889    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f6164d2a-4adb-4ef7-8443-d53e915eb70e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.115176 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.115158    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f6164d2a-4adb-4ef7-8443-d53e915eb70e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.115455 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.115434    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f6164d2a-4adb-4ef7-8443-d53e915eb70e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.116061 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.116030    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f6164d2a-4adb-4ef7-8443-d53e915eb70e-config-out\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.116662 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.116145    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f6164d2a-4adb-4ef7-8443-d53e915eb70e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.116662 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.116231    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f6164d2a-4adb-4ef7-8443-d53e915eb70e-web-config\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.116662 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.116372    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f6164d2a-4adb-4ef7-8443-d53e915eb70e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.116662 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.116410    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f6164d2a-4adb-4ef7-8443-d53e915eb70e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.116662 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.116612    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f6164d2a-4adb-4ef7-8443-d53e915eb70e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.116662 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.116619    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f6164d2a-4adb-4ef7-8443-d53e915eb70e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.117293 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.117274    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f6164d2a-4adb-4ef7-8443-d53e915eb70e-config\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.117658 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.117642    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f6164d2a-4adb-4ef7-8443-d53e915eb70e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.119776 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.119760    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c5gx\" (UniqueName: \"kubernetes.io/projected/f6164d2a-4adb-4ef7-8443-d53e915eb70e-kube-api-access-6c5gx\") pod \"prometheus-k8s-0\" (UID: \"f6164d2a-4adb-4ef7-8443-d53e915eb70e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.237813 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.237782    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:36.385668 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.385632    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 15 20:50:36.388494 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:50:36.388466    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6164d2a_4adb_4ef7_8443_d53e915eb70e.slice/crio-24f0fd00c2cb3b0aa52da67253f5d8c2126e587cf0e8228e2436e68f665c6475 WatchSource:0}: Error finding container 24f0fd00c2cb3b0aa52da67253f5d8c2126e587cf0e8228e2436e68f665c6475: Status 404 returned error can't find the container with id 24f0fd00c2cb3b0aa52da67253f5d8c2126e587cf0e8228e2436e68f665c6475
Jul 15 20:50:36.567882 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.567796    2566 generic.go:356] "Generic (PLEG): container finished" podID="f6164d2a-4adb-4ef7-8443-d53e915eb70e" containerID="d3eef3d38f235ad9facb70dab1eaef23a5c2e4b417e5d2c96dfc3ef1b0453627" exitCode=0
Jul 15 20:50:36.568025 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.567889    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f6164d2a-4adb-4ef7-8443-d53e915eb70e","Type":"ContainerDied","Data":"d3eef3d38f235ad9facb70dab1eaef23a5c2e4b417e5d2c96dfc3ef1b0453627"}
Jul 15 20:50:36.568025 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:36.568012    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f6164d2a-4adb-4ef7-8443-d53e915eb70e","Type":"ContainerStarted","Data":"24f0fd00c2cb3b0aa52da67253f5d8c2126e587cf0e8228e2436e68f665c6475"}
Jul 15 20:50:37.575439 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:37.575348    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f6164d2a-4adb-4ef7-8443-d53e915eb70e","Type":"ContainerStarted","Data":"412cbe029767330075e89bca9c956eb7f432abcd61f25a7c89d5dbb69ac2a40b"}
Jul 15 20:50:37.575779 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:37.575471    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f6164d2a-4adb-4ef7-8443-d53e915eb70e","Type":"ContainerStarted","Data":"682f59be4c05d4fc64b010f4ffd6082a418b9cfcda9fa42bdac48207dce0a2f7"}
Jul 15 20:50:37.575779 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:37.575512    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f6164d2a-4adb-4ef7-8443-d53e915eb70e","Type":"ContainerStarted","Data":"b5f96300185fc26a8ac42d5749bcc036bcc150edcbee0265168de19c857de5b0"}
Jul 15 20:50:37.575779 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:37.575551    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f6164d2a-4adb-4ef7-8443-d53e915eb70e","Type":"ContainerStarted","Data":"62882e243a71dbabe64a53a40dc9171be7eff7f59f33c680bb4121dd75d8d7a0"}
Jul 15 20:50:37.575779 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:37.575591    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f6164d2a-4adb-4ef7-8443-d53e915eb70e","Type":"ContainerStarted","Data":"c509ac70ab37aafe12c668c0fdacccb1726d41c2dc79790e2a75cb1ff01b859e"}
Jul 15 20:50:37.575779 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:37.575627    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f6164d2a-4adb-4ef7-8443-d53e915eb70e","Type":"ContainerStarted","Data":"983de7567cf502602cce1b900a2523aaaf48cad960c393b557ab516eb740affc"}
Jul 15 20:50:37.617080 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:37.617023    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.59945132 podStartE2EDuration="2.617007557s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.556237ms" isStatefulPod=false podCreationTimestamp="2026-07-15 20:50:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:50:37.61454738 +0000 UTC m=+170.763585438" watchObservedRunningTime="2026-07-15 20:50:37.617007557 +0000 UTC m=+170.766045614"
Jul 15 20:50:37.757909 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:37.757862    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aaa43101-957c-4064-92db-ba62aabdd8d9" path="/var/lib/kubelet/pods/aaa43101-957c-4064-92db-ba62aabdd8d9/volumes"
Jul 15 20:50:39.799383 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:39.799300    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:50:39.939645 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:39.939596    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:50:40.486768 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:40.486731    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:50:40.609878 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:40.609843    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:50:41.238491 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:41.238453    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:50:51.171946 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:51.171886    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:50:51.272370 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:51.272338    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:50:51.334936 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:50:51.334888    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:51:05.552912 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:51:05.552855    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:51:05.657328 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:51:05.657295    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:51:36.237990 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:51:36.237956    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:51:36.253051 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:51:36.253024    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:51:36.787757 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:51:36.787733    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 15 20:51:45.234366 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:51:45.234331    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:51:45.306470 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:51:45.306437    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:51:53.575506 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:51:53.575473    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:51:53.667613 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:51:53.667578    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:51:57.200984 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:51:57.200949    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:51:57.260947 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:51:57.260914    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:52:01.674502 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:52:01.674460    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:52:01.775866 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:52:01.775839    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:52:01.838089 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:52:01.838056    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:52:04.396437 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:52:04.396405    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:52:04.456581 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:52:04.456550    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:52:04.515225 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:52:04.515201    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:52:47.586612 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:52:47.586581    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-652lk_f9a159bb-39c4-4cca-92ff-1c74d739e311/ovn-acl-logging/0.log"
Jul 15 20:52:47.587316 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:52:47.587298    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-652lk_f9a159bb-39c4-4cca-92ff-1c74d739e311/ovn-acl-logging/0.log"
Jul 15 20:52:47.599000 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:52:47.598977    2566 kubelet.go:1648] "Image garbage collection succeeded"
Jul 15 20:52:57.491098 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:52:57.491063    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:52:58.691750 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:52:58.691708    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:53:35.221065 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:53:35.221029    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:53:36.601398 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:53:36.601355    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:53:39.062922 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:53:39.062819    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:53:39.162678 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:53:39.162646    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:53:40.070102 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:53:40.070065    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:53:40.203104 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:53:40.203068    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:53:41.096956 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:53:41.096921    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:53:41.206401 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:53:41.206366    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:53:43.480159 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:53:43.480114    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:53:43.751400 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:53:43.751315    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:53:46.059303 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:53:46.059273    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:53:46.158037 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:53:46.158006    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:53:51.314215 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:53:51.314186    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:53:53.461299 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:53:53.461261    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:53:53.568823 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:53:53.568792    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:53:58.609463 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:53:58.609429    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:53:58.707887 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:53:58.707847    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:54:00.923045 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:00.923016    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:54:16.142394 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:16.142364    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:54:16.231195 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:16.231155    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:54:16.713907 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:16.713872    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:54:16.813547 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:16.813513    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:54:16.916382 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:16.916352    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:54:17.133528 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:17.133450    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:54:17.349129 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:17.349094    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:54:17.454279 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:17.454249    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b849856cb-lcvdt"]
Jul 15 20:54:17.457467 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:17.457449    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-b849856cb-lcvdt"
Jul 15 20:54:17.464502 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:17.464484    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 15 20:54:17.464662 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:17.464643    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-d2vtb\""
Jul 15 20:54:17.468404 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:17.468386    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 15 20:54:17.477200 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:17.477182    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 15 20:54:17.483245 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:17.483230    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 15 20:54:17.517367 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:17.517342    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b849856cb-lcvdt"]
Jul 15 20:54:17.534181 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:17.534152    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c84kn\" (UniqueName: \"kubernetes.io/projected/97177182-de80-4d9c-94ff-23c3300d11ce-kube-api-access-c84kn\") pod \"opendatahub-operator-controller-manager-b849856cb-lcvdt\" (UID: \"97177182-de80-4d9c-94ff-23c3300d11ce\") " pod="opendatahub/opendatahub-operator-controller-manager-b849856cb-lcvdt"
Jul 15 20:54:17.534292 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:17.534187    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97177182-de80-4d9c-94ff-23c3300d11ce-tmp\") pod \"opendatahub-operator-controller-manager-b849856cb-lcvdt\" (UID: \"97177182-de80-4d9c-94ff-23c3300d11ce\") " pod="opendatahub/opendatahub-operator-controller-manager-b849856cb-lcvdt"
Jul 15 20:54:17.534292 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:17.534214    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/97177182-de80-4d9c-94ff-23c3300d11ce-webhook-cert\") pod \"opendatahub-operator-controller-manager-b849856cb-lcvdt\" (UID: \"97177182-de80-4d9c-94ff-23c3300d11ce\") " pod="opendatahub/opendatahub-operator-controller-manager-b849856cb-lcvdt"
Jul 15 20:54:17.534292 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:17.534282    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/97177182-de80-4d9c-94ff-23c3300d11ce-manifests\") pod \"opendatahub-operator-controller-manager-b849856cb-lcvdt\" (UID: \"97177182-de80-4d9c-94ff-23c3300d11ce\") " pod="opendatahub/opendatahub-operator-controller-manager-b849856cb-lcvdt"
Jul 15 20:54:17.534403 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:17.534315    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/97177182-de80-4d9c-94ff-23c3300d11ce-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b849856cb-lcvdt\" (UID: \"97177182-de80-4d9c-94ff-23c3300d11ce\") " pod="opendatahub/opendatahub-operator-controller-manager-b849856cb-lcvdt"
Jul 15 20:54:17.635664 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:17.635630    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c84kn\" (UniqueName: \"kubernetes.io/projected/97177182-de80-4d9c-94ff-23c3300d11ce-kube-api-access-c84kn\") pod \"opendatahub-operator-controller-manager-b849856cb-lcvdt\" (UID: \"97177182-de80-4d9c-94ff-23c3300d11ce\") " pod="opendatahub/opendatahub-operator-controller-manager-b849856cb-lcvdt"
Jul 15 20:54:17.635664 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:17.635667    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97177182-de80-4d9c-94ff-23c3300d11ce-tmp\") pod \"opendatahub-operator-controller-manager-b849856cb-lcvdt\" (UID: \"97177182-de80-4d9c-94ff-23c3300d11ce\") " pod="opendatahub/opendatahub-operator-controller-manager-b849856cb-lcvdt"
Jul 15 20:54:17.635879 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:17.635692    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/97177182-de80-4d9c-94ff-23c3300d11ce-webhook-cert\") pod \"opendatahub-operator-controller-manager-b849856cb-lcvdt\" (UID: \"97177182-de80-4d9c-94ff-23c3300d11ce\") " pod="opendatahub/opendatahub-operator-controller-manager-b849856cb-lcvdt"
Jul 15 20:54:17.635879 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:17.635716    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/97177182-de80-4d9c-94ff-23c3300d11ce-manifests\") pod \"opendatahub-operator-controller-manager-b849856cb-lcvdt\" (UID: \"97177182-de80-4d9c-94ff-23c3300d11ce\") " pod="opendatahub/opendatahub-operator-controller-manager-b849856cb-lcvdt"
Jul 15 20:54:17.635879 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:17.635750    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/97177182-de80-4d9c-94ff-23c3300d11ce-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b849856cb-lcvdt\" (UID: \"97177182-de80-4d9c-94ff-23c3300d11ce\") " pod="opendatahub/opendatahub-operator-controller-manager-b849856cb-lcvdt"
Jul 15 20:54:17.636262 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:17.636238    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/97177182-de80-4d9c-94ff-23c3300d11ce-manifests\") pod \"opendatahub-operator-controller-manager-b849856cb-lcvdt\" (UID: \"97177182-de80-4d9c-94ff-23c3300d11ce\") " pod="opendatahub/opendatahub-operator-controller-manager-b849856cb-lcvdt"
Jul 15 20:54:17.638240 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:17.638209    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97177182-de80-4d9c-94ff-23c3300d11ce-tmp\") pod \"opendatahub-operator-controller-manager-b849856cb-lcvdt\" (UID: \"97177182-de80-4d9c-94ff-23c3300d11ce\") " pod="opendatahub/opendatahub-operator-controller-manager-b849856cb-lcvdt"
Jul 15 20:54:17.638456 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:17.638434    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/97177182-de80-4d9c-94ff-23c3300d11ce-webhook-cert\") pod \"opendatahub-operator-controller-manager-b849856cb-lcvdt\" (UID: \"97177182-de80-4d9c-94ff-23c3300d11ce\") " pod="opendatahub/opendatahub-operator-controller-manager-b849856cb-lcvdt"
Jul 15 20:54:17.638531 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:17.638505    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/97177182-de80-4d9c-94ff-23c3300d11ce-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b849856cb-lcvdt\" (UID: \"97177182-de80-4d9c-94ff-23c3300d11ce\") " pod="opendatahub/opendatahub-operator-controller-manager-b849856cb-lcvdt"
Jul 15 20:54:17.649205 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:17.649178    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c84kn\" (UniqueName: \"kubernetes.io/projected/97177182-de80-4d9c-94ff-23c3300d11ce-kube-api-access-c84kn\") pod \"opendatahub-operator-controller-manager-b849856cb-lcvdt\" (UID: \"97177182-de80-4d9c-94ff-23c3300d11ce\") " pod="opendatahub/opendatahub-operator-controller-manager-b849856cb-lcvdt"
Jul 15 20:54:17.657874 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:17.657850    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:54:17.757342 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:17.757252    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:54:17.767207 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:17.767186    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-b849856cb-lcvdt"
Jul 15 20:54:17.847057 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:17.847025    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:54:17.919505 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:17.919460    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:54:17.921023 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:17.920914    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b849856cb-lcvdt"]
Jul 15 20:54:17.923351 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:54:17.923319    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97177182_de80_4d9c_94ff_23c3300d11ce.slice/crio-a3c44b16735f14214c7d2e11fb91be20fc6690c04f7cfec5b3de9f5e0512880c WatchSource:0}: Error finding container a3c44b16735f14214c7d2e11fb91be20fc6690c04f7cfec5b3de9f5e0512880c: Status 404 returned error can't find the container with id a3c44b16735f14214c7d2e11fb91be20fc6690c04f7cfec5b3de9f5e0512880c
Jul 15 20:54:17.925107 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:17.925089    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 15 20:54:18.013245 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:18.013164    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:54:18.113739 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:18.113707    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:54:18.214791 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:18.214761    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:54:18.308623 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:18.308532    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b849856cb-lcvdt" event={"ID":"97177182-de80-4d9c-94ff-23c3300d11ce","Type":"ContainerStarted","Data":"a3c44b16735f14214c7d2e11fb91be20fc6690c04f7cfec5b3de9f5e0512880c"}
Jul 15 20:54:21.247222 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:21.247133    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:54:21.325611 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:21.325572    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b849856cb-lcvdt" event={"ID":"97177182-de80-4d9c-94ff-23c3300d11ce","Type":"ContainerStarted","Data":"02db148f754960fc3a21ac791f3dc88b9561ef7557bffe9bec84dd675654fd8a"}
Jul 15 20:54:21.368041 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:21.368010    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:54:22.331114 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:22.331078    2566 generic.go:356] "Generic (PLEG): container finished" podID="97177182-de80-4d9c-94ff-23c3300d11ce" containerID="02db148f754960fc3a21ac791f3dc88b9561ef7557bffe9bec84dd675654fd8a" exitCode=0
Jul 15 20:54:22.331535 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:22.331169    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b849856cb-lcvdt" event={"ID":"97177182-de80-4d9c-94ff-23c3300d11ce","Type":"ContainerDied","Data":"02db148f754960fc3a21ac791f3dc88b9561ef7557bffe9bec84dd675654fd8a"}
Jul 15 20:54:23.336693 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:23.336660    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b849856cb-lcvdt" event={"ID":"97177182-de80-4d9c-94ff-23c3300d11ce","Type":"ContainerStarted","Data":"7679bc94d18985c72e885f369f063d730b77a3a0e984afbb01e35e626f242a26"}
Jul 15 20:54:23.337066 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:23.336799    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-b849856cb-lcvdt"
Jul 15 20:54:23.404064 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:23.404014    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-b849856cb-lcvdt" podStartSLOduration=2.003282689 podStartE2EDuration="6.403997789s" totalImagesPullingTime="3.321497042s" totalInitContainerRuntime="1.079218058s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:54:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:54:23.403660816 +0000 UTC m=+396.552698896" watchObservedRunningTime="2026-07-15 20:54:23.403997789 +0000 UTC m=+396.553035844"
Jul 15 20:54:32.686994 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:32.686924    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:54:32.787935 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:32.787879    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:54:34.342752 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:34.342727    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-b849856cb-lcvdt"
Jul 15 20:54:36.644627 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:36.644593    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:54:36.793337 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:36.793304    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:54:37.311662 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:37.311626    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:54:37.469094 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:37.469053    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:54:37.819416 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:37.819379    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:54:37.908200 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:37.908163    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:54:37.990500 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:37.990465    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:54:38.372018 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:38.371984    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:54:38.783262 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:38.783229    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:54:39.178415 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:39.178308    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:54:39.288484 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:39.288430    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:54:39.465265 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:39.465236    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:54:40.540876 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:40.540838    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:54:40.658831 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:40.658794    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:54:40.874317 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:40.874232    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:54:43.998456 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:43.998420    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:54:44.100847 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:44.100810    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:54:44.845366 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:44.845329    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:54:45.157116 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:45.157037    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:54:45.333253 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:45.333216    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:54:45.625436 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:45.625406    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6bd6785cdc-s9bmx"]
Jul 15 20:54:45.628537 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:45.628517    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6bd6785cdc-s9bmx"
Jul 15 20:54:45.631771 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:45.631751    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 15 20:54:45.633183 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:45.633153    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 15 20:54:45.633285 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:45.633183    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 15 20:54:45.633285 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:45.633207    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 15 20:54:45.633285 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:45.633257    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 15 20:54:45.633563 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:45.633548    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-v649g\""
Jul 15 20:54:45.641272 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:45.641251    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6bd6785cdc-s9bmx"]
Jul 15 20:54:45.772297 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:45.772266    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5557b001-8b97-4d1e-a501-fdb11e14c1c4-manager-config\") pod \"lws-controller-manager-6bd6785cdc-s9bmx\" (UID: \"5557b001-8b97-4d1e-a501-fdb11e14c1c4\") " pod="openshift-lws-operator/lws-controller-manager-6bd6785cdc-s9bmx"
Jul 15 20:54:45.772475 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:45.772303    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwrmx\" (UniqueName: \"kubernetes.io/projected/5557b001-8b97-4d1e-a501-fdb11e14c1c4-kube-api-access-wwrmx\") pod \"lws-controller-manager-6bd6785cdc-s9bmx\" (UID: \"5557b001-8b97-4d1e-a501-fdb11e14c1c4\") " pod="openshift-lws-operator/lws-controller-manager-6bd6785cdc-s9bmx"
Jul 15 20:54:45.772475 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:45.772325    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5557b001-8b97-4d1e-a501-fdb11e14c1c4-cert\") pod \"lws-controller-manager-6bd6785cdc-s9bmx\" (UID: \"5557b001-8b97-4d1e-a501-fdb11e14c1c4\") " pod="openshift-lws-operator/lws-controller-manager-6bd6785cdc-s9bmx"
Jul 15 20:54:45.772475 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:45.772403    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5557b001-8b97-4d1e-a501-fdb11e14c1c4-metrics-cert\") pod \"lws-controller-manager-6bd6785cdc-s9bmx\" (UID: \"5557b001-8b97-4d1e-a501-fdb11e14c1c4\") " pod="openshift-lws-operator/lws-controller-manager-6bd6785cdc-s9bmx"
Jul 15 20:54:45.873184 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:45.873132    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwrmx\" (UniqueName: \"kubernetes.io/projected/5557b001-8b97-4d1e-a501-fdb11e14c1c4-kube-api-access-wwrmx\") pod \"lws-controller-manager-6bd6785cdc-s9bmx\" (UID: \"5557b001-8b97-4d1e-a501-fdb11e14c1c4\") " pod="openshift-lws-operator/lws-controller-manager-6bd6785cdc-s9bmx"
Jul 15 20:54:45.873184 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:45.873188    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5557b001-8b97-4d1e-a501-fdb11e14c1c4-cert\") pod \"lws-controller-manager-6bd6785cdc-s9bmx\" (UID: \"5557b001-8b97-4d1e-a501-fdb11e14c1c4\") " pod="openshift-lws-operator/lws-controller-manager-6bd6785cdc-s9bmx"
Jul 15 20:54:45.873455 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:45.873224    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5557b001-8b97-4d1e-a501-fdb11e14c1c4-metrics-cert\") pod \"lws-controller-manager-6bd6785cdc-s9bmx\" (UID: \"5557b001-8b97-4d1e-a501-fdb11e14c1c4\") " pod="openshift-lws-operator/lws-controller-manager-6bd6785cdc-s9bmx"
Jul 15 20:54:45.873455 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:45.873279    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5557b001-8b97-4d1e-a501-fdb11e14c1c4-manager-config\") pod \"lws-controller-manager-6bd6785cdc-s9bmx\" (UID: \"5557b001-8b97-4d1e-a501-fdb11e14c1c4\") " pod="openshift-lws-operator/lws-controller-manager-6bd6785cdc-s9bmx"
Jul 15 20:54:45.874007 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:45.873981    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5557b001-8b97-4d1e-a501-fdb11e14c1c4-manager-config\") pod \"lws-controller-manager-6bd6785cdc-s9bmx\" (UID: \"5557b001-8b97-4d1e-a501-fdb11e14c1c4\") " pod="openshift-lws-operator/lws-controller-manager-6bd6785cdc-s9bmx"
Jul 15 20:54:45.875734 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:45.875670    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5557b001-8b97-4d1e-a501-fdb11e14c1c4-metrics-cert\") pod \"lws-controller-manager-6bd6785cdc-s9bmx\" (UID: \"5557b001-8b97-4d1e-a501-fdb11e14c1c4\") " pod="openshift-lws-operator/lws-controller-manager-6bd6785cdc-s9bmx"
Jul 15 20:54:45.875734 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:45.875696    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5557b001-8b97-4d1e-a501-fdb11e14c1c4-cert\") pod \"lws-controller-manager-6bd6785cdc-s9bmx\" (UID: \"5557b001-8b97-4d1e-a501-fdb11e14c1c4\") " pod="openshift-lws-operator/lws-controller-manager-6bd6785cdc-s9bmx"
Jul 15 20:54:45.883607 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:45.883581    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwrmx\" (UniqueName: \"kubernetes.io/projected/5557b001-8b97-4d1e-a501-fdb11e14c1c4-kube-api-access-wwrmx\") pod \"lws-controller-manager-6bd6785cdc-s9bmx\" (UID: \"5557b001-8b97-4d1e-a501-fdb11e14c1c4\") " pod="openshift-lws-operator/lws-controller-manager-6bd6785cdc-s9bmx"
Jul 15 20:54:45.938447 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:45.938413    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6bd6785cdc-s9bmx"
Jul 15 20:54:46.116469 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:46.116441    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6bd6785cdc-s9bmx"]
Jul 15 20:54:46.117411 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:54:46.117389    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5557b001_8b97_4d1e_a501_fdb11e14c1c4.slice/crio-85bb6ff38b3fe78b3b8b0efbff2b5357c3b76a615f34095ad845fd2131ef9a56 WatchSource:0}: Error finding container 85bb6ff38b3fe78b3b8b0efbff2b5357c3b76a615f34095ad845fd2131ef9a56: Status 404 returned error can't find the container with id 85bb6ff38b3fe78b3b8b0efbff2b5357c3b76a615f34095ad845fd2131ef9a56
Jul 15 20:54:46.420717 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:46.420684    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6bd6785cdc-s9bmx" event={"ID":"5557b001-8b97-4d1e-a501-fdb11e14c1c4","Type":"ContainerStarted","Data":"85bb6ff38b3fe78b3b8b0efbff2b5357c3b76a615f34095ad845fd2131ef9a56"}
Jul 15 20:54:48.370558 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:48.370478    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:54:48.448946 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:48.447816    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-xlrk4"]
Jul 15 20:54:48.451480 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:48.451454    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-kcwtz"]
Jul 15 20:54:48.451631 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:48.451586    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xlrk4"
Jul 15 20:54:48.454529 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:48.454503    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 15 20:54:48.454693 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:48.454670    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kcwtz"
Jul 15 20:54:48.455054 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:48.455036    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-sxhkg\""
Jul 15 20:54:48.458208 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:48.458190    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-jrbrj\""
Jul 15 20:54:48.458466 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:48.458447    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 15 20:54:48.469246 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:48.469190    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-xlrk4"]
Jul 15 20:54:48.472740 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:48.472717    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-kcwtz"]
Jul 15 20:54:48.505399 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:48.505355    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4bc02df2-6841-4cda-b064-a19ea5f10523-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-xlrk4\" (UID: \"4bc02df2-6841-4cda-b064-a19ea5f10523\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xlrk4"
Jul 15 20:54:48.505593 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:48.505448    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4a0582f-65dd-47c9-91a1-e12bee5fcc56-cert\") pod \"kserve-controller-manager-d6f4b65b8-kcwtz\" (UID: \"f4a0582f-65dd-47c9-91a1-e12bee5fcc56\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kcwtz"
Jul 15 20:54:48.505593 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:48.505477    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnrdm\" (UniqueName: \"kubernetes.io/projected/f4a0582f-65dd-47c9-91a1-e12bee5fcc56-kube-api-access-bnrdm\") pod \"kserve-controller-manager-d6f4b65b8-kcwtz\" (UID: \"f4a0582f-65dd-47c9-91a1-e12bee5fcc56\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kcwtz"
Jul 15 20:54:48.505593 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:48.505526    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgmcj\" (UniqueName: \"kubernetes.io/projected/4bc02df2-6841-4cda-b064-a19ea5f10523-kube-api-access-tgmcj\") pod \"llmisvc-controller-manager-777c6bd7d5-xlrk4\" (UID: \"4bc02df2-6841-4cda-b064-a19ea5f10523\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xlrk4"
Jul 15 20:54:48.505840 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:48.505788    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:54:48.606036 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:48.606007    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgmcj\" (UniqueName: \"kubernetes.io/projected/4bc02df2-6841-4cda-b064-a19ea5f10523-kube-api-access-tgmcj\") pod \"llmisvc-controller-manager-777c6bd7d5-xlrk4\" (UID: \"4bc02df2-6841-4cda-b064-a19ea5f10523\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xlrk4"
Jul 15 20:54:48.606214 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:48.606051    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4bc02df2-6841-4cda-b064-a19ea5f10523-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-xlrk4\" (UID: \"4bc02df2-6841-4cda-b064-a19ea5f10523\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xlrk4"
Jul 15 20:54:48.606214 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:48.606097    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4a0582f-65dd-47c9-91a1-e12bee5fcc56-cert\") pod \"kserve-controller-manager-d6f4b65b8-kcwtz\" (UID: \"f4a0582f-65dd-47c9-91a1-e12bee5fcc56\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kcwtz"
Jul 15 20:54:48.606214 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:48.606120    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnrdm\" (UniqueName: \"kubernetes.io/projected/f4a0582f-65dd-47c9-91a1-e12bee5fcc56-kube-api-access-bnrdm\") pod \"kserve-controller-manager-d6f4b65b8-kcwtz\" (UID: \"f4a0582f-65dd-47c9-91a1-e12bee5fcc56\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kcwtz"
Jul 15 20:54:48.606377 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:54:48.606224    2566 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 15 20:54:48.606377 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:54:48.606288    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bc02df2-6841-4cda-b064-a19ea5f10523-cert podName:4bc02df2-6841-4cda-b064-a19ea5f10523 nodeName:}" failed. No retries permitted until 2026-07-15 20:54:49.106267632 +0000 UTC m=+422.255305682 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/4bc02df2-6841-4cda-b064-a19ea5f10523-cert") pod "llmisvc-controller-manager-777c6bd7d5-xlrk4" (UID: "4bc02df2-6841-4cda-b064-a19ea5f10523") : secret "llmisvc-webhook-server-cert" not found
Jul 15 20:54:48.606377 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:54:48.606228    2566 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 15 20:54:48.606519 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:54:48.606390    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4a0582f-65dd-47c9-91a1-e12bee5fcc56-cert podName:f4a0582f-65dd-47c9-91a1-e12bee5fcc56 nodeName:}" failed. No retries permitted until 2026-07-15 20:54:49.106366046 +0000 UTC m=+422.255404097 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f4a0582f-65dd-47c9-91a1-e12bee5fcc56-cert") pod "kserve-controller-manager-d6f4b65b8-kcwtz" (UID: "f4a0582f-65dd-47c9-91a1-e12bee5fcc56") : secret "kserve-webhook-server-cert" not found
Jul 15 20:54:48.622724 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:48.622658    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgmcj\" (UniqueName: \"kubernetes.io/projected/4bc02df2-6841-4cda-b064-a19ea5f10523-kube-api-access-tgmcj\") pod \"llmisvc-controller-manager-777c6bd7d5-xlrk4\" (UID: \"4bc02df2-6841-4cda-b064-a19ea5f10523\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xlrk4"
Jul 15 20:54:48.627393 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:48.627359    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnrdm\" (UniqueName: \"kubernetes.io/projected/f4a0582f-65dd-47c9-91a1-e12bee5fcc56-kube-api-access-bnrdm\") pod \"kserve-controller-manager-d6f4b65b8-kcwtz\" (UID: \"f4a0582f-65dd-47c9-91a1-e12bee5fcc56\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kcwtz"
Jul 15 20:54:48.699482 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:48.699462    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:54:48.809042 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:48.808978    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:54:48.891705 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:48.891597    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:54:49.108916 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:49.108861    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4bc02df2-6841-4cda-b064-a19ea5f10523-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-xlrk4\" (UID: \"4bc02df2-6841-4cda-b064-a19ea5f10523\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xlrk4"
Jul 15 20:54:49.109096 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:49.108944    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4a0582f-65dd-47c9-91a1-e12bee5fcc56-cert\") pod \"kserve-controller-manager-d6f4b65b8-kcwtz\" (UID: \"f4a0582f-65dd-47c9-91a1-e12bee5fcc56\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kcwtz"
Jul 15 20:54:49.109096 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:54:49.109010    2566 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 15 20:54:49.109096 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:54:49.109088    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bc02df2-6841-4cda-b064-a19ea5f10523-cert podName:4bc02df2-6841-4cda-b064-a19ea5f10523 nodeName:}" failed. No retries permitted until 2026-07-15 20:54:50.109070437 +0000 UTC m=+423.258108472 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/4bc02df2-6841-4cda-b064-a19ea5f10523-cert") pod "llmisvc-controller-manager-777c6bd7d5-xlrk4" (UID: "4bc02df2-6841-4cda-b064-a19ea5f10523") : secret "llmisvc-webhook-server-cert" not found
Jul 15 20:54:49.111563 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:49.111540    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4a0582f-65dd-47c9-91a1-e12bee5fcc56-cert\") pod \"kserve-controller-manager-d6f4b65b8-kcwtz\" (UID: \"f4a0582f-65dd-47c9-91a1-e12bee5fcc56\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kcwtz"
Jul 15 20:54:49.326599 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:49.326545    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:54:49.375614 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:49.375587    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kcwtz"
Jul 15 20:54:49.438028 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:49.437613    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6bd6785cdc-s9bmx" event={"ID":"5557b001-8b97-4d1e-a501-fdb11e14c1c4","Type":"ContainerStarted","Data":"3bab0f89d71e8f3d884f61470f7f811c2f13ea8d7af39e82195396b1f04ded40"}
Jul 15 20:54:49.438028 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:49.437734    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6bd6785cdc-s9bmx"
Jul 15 20:54:49.472427 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:49.472114    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6bd6785cdc-s9bmx" podStartSLOduration=1.9088032350000002 podStartE2EDuration="4.472092689s" totalImagesPullingTime="2.563289454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:54:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:54:49.471686594 +0000 UTC m=+422.620724648" watchObservedRunningTime="2026-07-15 20:54:49.472092689 +0000 UTC m=+422.621130746"
Jul 15 20:54:49.504326 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:49.504302    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-kcwtz"]
Jul 15 20:54:49.506497 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:54:49.506474    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4a0582f_65dd_47c9_91a1_e12bee5fcc56.slice/crio-51645f573f6dbe7fb36938abcd2db4a2118d49f8f6429aa892cc53eaa8723c15 WatchSource:0}: Error finding container 51645f573f6dbe7fb36938abcd2db4a2118d49f8f6429aa892cc53eaa8723c15: Status 404 returned error can't find the container with id 51645f573f6dbe7fb36938abcd2db4a2118d49f8f6429aa892cc53eaa8723c15
Jul 15 20:54:50.117170 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:50.117129    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4bc02df2-6841-4cda-b064-a19ea5f10523-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-xlrk4\" (UID: \"4bc02df2-6841-4cda-b064-a19ea5f10523\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xlrk4"
Jul 15 20:54:50.119662 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:50.119643    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4bc02df2-6841-4cda-b064-a19ea5f10523-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-xlrk4\" (UID: \"4bc02df2-6841-4cda-b064-a19ea5f10523\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xlrk4"
Jul 15 20:54:50.266056 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:50.266020    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xlrk4"
Jul 15 20:54:50.425641 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:50.425615    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-xlrk4"]
Jul 15 20:54:50.428639 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:54:50.428604    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4bc02df2_6841_4cda_b064_a19ea5f10523.slice/crio-cfe6b6aa9d3bfdb275c1341b36472f98ccbe236aa12995421c83cf33ea8e6890 WatchSource:0}: Error finding container cfe6b6aa9d3bfdb275c1341b36472f98ccbe236aa12995421c83cf33ea8e6890: Status 404 returned error can't find the container with id cfe6b6aa9d3bfdb275c1341b36472f98ccbe236aa12995421c83cf33ea8e6890
Jul 15 20:54:50.443089 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:50.443063    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xlrk4" event={"ID":"4bc02df2-6841-4cda-b064-a19ea5f10523","Type":"ContainerStarted","Data":"cfe6b6aa9d3bfdb275c1341b36472f98ccbe236aa12995421c83cf33ea8e6890"}
Jul 15 20:54:50.444274 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:50.444254    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kcwtz" event={"ID":"f4a0582f-65dd-47c9-91a1-e12bee5fcc56","Type":"ContainerStarted","Data":"51645f573f6dbe7fb36938abcd2db4a2118d49f8f6429aa892cc53eaa8723c15"}
Jul 15 20:54:52.466453 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:52.466408    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kcwtz" event={"ID":"f4a0582f-65dd-47c9-91a1-e12bee5fcc56","Type":"ContainerStarted","Data":"a632ecc6038f92aa47055e4e30e06ca3969364bcbe0712b773a27ff180328d82"}
Jul 15 20:54:52.466918 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:52.466554    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kcwtz"
Jul 15 20:54:52.495251 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:52.495145    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kcwtz" podStartSLOduration=1.68959445 podStartE2EDuration="4.495127123s" totalImagesPullingTime="2.805532673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:54:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:54:52.493844802 +0000 UTC m=+425.642882880" watchObservedRunningTime="2026-07-15 20:54:52.495127123 +0000 UTC m=+425.644165179"
Jul 15 20:54:53.471555 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:53.471517    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xlrk4" event={"ID":"4bc02df2-6841-4cda-b064-a19ea5f10523","Type":"ContainerStarted","Data":"cda77f6e7bea62f5419613df7cbf96108dc3ba66d93799483e64881904a037a5"}
Jul 15 20:54:53.472064 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:53.471775    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xlrk4"
Jul 15 20:54:53.495379 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:54:53.495329    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xlrk4" podStartSLOduration=2.825422231 podStartE2EDuration="5.495313747s" totalImagesPullingTime="2.669891516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:54:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:54:53.493621537 +0000 UTC m=+426.642659595" watchObservedRunningTime="2026-07-15 20:54:53.495313747 +0000 UTC m=+426.644351824"
Jul 15 20:55:00.447032 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:55:00.446999    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6bd6785cdc-s9bmx"
Jul 15 20:55:23.477008 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:55:23.476923    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kcwtz"
Jul 15 20:55:24.477685 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:55:24.477653    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xlrk4"
Jul 15 20:55:24.966057 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:55:24.966012    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:55:27.654453 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:55:27.654412    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:55:38.189862 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:55:38.189800    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:55:38.278465 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:55:38.278409    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:55:38.403873 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:55:38.403839    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:55:38.920286 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:55:38.920253    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-24hbb"]
Jul 15 20:55:38.926249 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:55:38.926229    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-24hbb"
Jul 15 20:55:38.930932 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:55:38.930890    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-6ztg9\""
Jul 15 20:55:38.931708 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:55:38.931689    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 15 20:55:38.932099 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:55:38.932080    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 15 20:55:38.948682 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:55:38.948654    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-24hbb"]
Jul 15 20:55:39.018576 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:55:39.018540    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgrvq\" (UniqueName: \"kubernetes.io/projected/f75b800f-3428-4734-9e43-98fedc98e8f5-kube-api-access-lgrvq\") pod \"kuadrant-operator-catalog-24hbb\" (UID: \"f75b800f-3428-4734-9e43-98fedc98e8f5\") " pod="kuadrant-system/kuadrant-operator-catalog-24hbb"
Jul 15 20:55:39.119347 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:55:39.119308    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgrvq\" (UniqueName: \"kubernetes.io/projected/f75b800f-3428-4734-9e43-98fedc98e8f5-kube-api-access-lgrvq\") pod \"kuadrant-operator-catalog-24hbb\" (UID: \"f75b800f-3428-4734-9e43-98fedc98e8f5\") " pod="kuadrant-system/kuadrant-operator-catalog-24hbb"
Jul 15 20:55:39.139963 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:55:39.139929    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgrvq\" (UniqueName: \"kubernetes.io/projected/f75b800f-3428-4734-9e43-98fedc98e8f5-kube-api-access-lgrvq\") pod \"kuadrant-operator-catalog-24hbb\" (UID: \"f75b800f-3428-4734-9e43-98fedc98e8f5\") " pod="kuadrant-system/kuadrant-operator-catalog-24hbb"
Jul 15 20:55:39.238846 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:55:39.238798    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-24hbb"
Jul 15 20:55:39.368713 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:55:39.368608    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-24hbb"]
Jul 15 20:55:39.370725 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:55:39.370694    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf75b800f_3428_4734_9e43_98fedc98e8f5.slice/crio-f24385867ae36c450afb6a02abae1c3e595f9fd19bf1e4706895a900dd13df2f WatchSource:0}: Error finding container f24385867ae36c450afb6a02abae1c3e595f9fd19bf1e4706895a900dd13df2f: Status 404 returned error can't find the container with id f24385867ae36c450afb6a02abae1c3e595f9fd19bf1e4706895a900dd13df2f
Jul 15 20:55:39.643633 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:55:39.643551    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-24hbb" event={"ID":"f75b800f-3428-4734-9e43-98fedc98e8f5","Type":"ContainerStarted","Data":"f24385867ae36c450afb6a02abae1c3e595f9fd19bf1e4706895a900dd13df2f"}
Jul 15 20:55:41.652707 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:55:41.652619    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-24hbb" event={"ID":"f75b800f-3428-4734-9e43-98fedc98e8f5","Type":"ContainerStarted","Data":"426e8fee17ec4f2ec843d2c89684e4165f0e23dea02a6eefe96bbe5fcb25d347"}
Jul 15 20:55:41.676385 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:55:41.676338    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-24hbb" podStartSLOduration=1.755028005 podStartE2EDuration="3.676322541s" totalImagesPullingTime="1.921294536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:55:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:55:41.676091287 +0000 UTC m=+474.825129345" watchObservedRunningTime="2026-07-15 20:55:41.676322541 +0000 UTC m=+474.825360597"
Jul 15 20:55:49.239124 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:55:49.239073    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-24hbb"
Jul 15 20:55:49.239124 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:55:49.239143    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-24hbb"
Jul 15 20:55:49.261247 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:55:49.261221    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-24hbb"
Jul 15 20:55:49.702251 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:55:49.702224    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-24hbb"
Jul 15 20:55:59.157768 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:55:59.157705    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:55:59.259791 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:55:59.259727    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:55:59.359861 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:55:59.359829    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:55:59.883955 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:55:59.883889    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:55:59.984529 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:55:59.984498    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:56:02.455862 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:02.455829    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:56:05.845696 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:05.845668    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:56:06.116090 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:06.116009    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:56:10.164446 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:10.164413    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t4rtd"]
Jul 15 20:56:10.167604 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:10.167585    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t4rtd"
Jul 15 20:56:10.172278 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:10.172253    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-k9x7l\""
Jul 15 20:56:10.187670 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:10.187639    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t4rtd"]
Jul 15 20:56:10.278425 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:10.278395    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/08bb52d6-c6ee-4072-985f-9891358602de-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-t4rtd\" (UID: \"08bb52d6-c6ee-4072-985f-9891358602de\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t4rtd"
Jul 15 20:56:10.278425 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:10.278428    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwz8t\" (UniqueName: \"kubernetes.io/projected/08bb52d6-c6ee-4072-985f-9891358602de-kube-api-access-jwz8t\") pod \"kuadrant-operator-controller-manager-8684c9ff49-t4rtd\" (UID: \"08bb52d6-c6ee-4072-985f-9891358602de\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t4rtd"
Jul 15 20:56:10.379137 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:10.379105    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/08bb52d6-c6ee-4072-985f-9891358602de-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-t4rtd\" (UID: \"08bb52d6-c6ee-4072-985f-9891358602de\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t4rtd"
Jul 15 20:56:10.379137 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:10.379141    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwz8t\" (UniqueName: \"kubernetes.io/projected/08bb52d6-c6ee-4072-985f-9891358602de-kube-api-access-jwz8t\") pod \"kuadrant-operator-controller-manager-8684c9ff49-t4rtd\" (UID: \"08bb52d6-c6ee-4072-985f-9891358602de\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t4rtd"
Jul 15 20:56:10.379529 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:10.379505    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/08bb52d6-c6ee-4072-985f-9891358602de-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-t4rtd\" (UID: \"08bb52d6-c6ee-4072-985f-9891358602de\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t4rtd"
Jul 15 20:56:10.398737 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:10.398702    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwz8t\" (UniqueName: \"kubernetes.io/projected/08bb52d6-c6ee-4072-985f-9891358602de-kube-api-access-jwz8t\") pod \"kuadrant-operator-controller-manager-8684c9ff49-t4rtd\" (UID: \"08bb52d6-c6ee-4072-985f-9891358602de\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t4rtd"
Jul 15 20:56:10.477435 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:10.477401    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t4rtd"
Jul 15 20:56:10.641290 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:10.641257    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t4rtd"]
Jul 15 20:56:10.645212 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:56:10.645181    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08bb52d6_c6ee_4072_985f_9891358602de.slice/crio-bc539d544f95f8ea41f386bc4911143c2e941c91c943b98cbb9cb964eda58828 WatchSource:0}: Error finding container bc539d544f95f8ea41f386bc4911143c2e941c91c943b98cbb9cb964eda58828: Status 404 returned error can't find the container with id bc539d544f95f8ea41f386bc4911143c2e941c91c943b98cbb9cb964eda58828
Jul 15 20:56:10.681336 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:10.681306    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:56:10.760352 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:10.760272    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t4rtd" event={"ID":"08bb52d6-c6ee-4072-985f-9891358602de","Type":"ContainerStarted","Data":"bc539d544f95f8ea41f386bc4911143c2e941c91c943b98cbb9cb964eda58828"}
Jul 15 20:56:10.789430 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:10.789396    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:56:10.885060 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:10.885026    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:56:11.064033 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:11.063948    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:56:12.578945 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:12.578913    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:56:12.982261 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:12.982232    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-bdvjc"]
Jul 15 20:56:12.985511 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:12.985492    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bdvjc"
Jul 15 20:56:12.991274 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:12.991252    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-kdsk7\""
Jul 15 20:56:13.002416 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:13.002385    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-bdvjc"]
Jul 15 20:56:13.101546 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:13.101512    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z82zx\" (UniqueName: \"kubernetes.io/projected/e5286d79-383d-4b78-a2b3-b8b6fea1b534-kube-api-access-z82zx\") pod \"authorino-operator-75cc8d5cb7-bdvjc\" (UID: \"e5286d79-383d-4b78-a2b3-b8b6fea1b534\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-bdvjc"
Jul 15 20:56:13.202955 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:13.202919    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z82zx\" (UniqueName: \"kubernetes.io/projected/e5286d79-383d-4b78-a2b3-b8b6fea1b534-kube-api-access-z82zx\") pod \"authorino-operator-75cc8d5cb7-bdvjc\" (UID: \"e5286d79-383d-4b78-a2b3-b8b6fea1b534\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-bdvjc"
Jul 15 20:56:13.218890 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:13.218856    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z82zx\" (UniqueName: \"kubernetes.io/projected/e5286d79-383d-4b78-a2b3-b8b6fea1b534-kube-api-access-z82zx\") pod \"authorino-operator-75cc8d5cb7-bdvjc\" (UID: \"e5286d79-383d-4b78-a2b3-b8b6fea1b534\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-bdvjc"
Jul 15 20:56:13.296760 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:13.296676    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bdvjc"
Jul 15 20:56:13.452433 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:13.452286    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-bdvjc"]
Jul 15 20:56:13.455058 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:56:13.455024    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5286d79_383d_4b78_a2b3_b8b6fea1b534.slice/crio-aa495cc752eeb7e096074b01d16d6605fb961f3ea3f4f3e3407ae2c4887d05cf WatchSource:0}: Error finding container aa495cc752eeb7e096074b01d16d6605fb961f3ea3f4f3e3407ae2c4887d05cf: Status 404 returned error can't find the container with id aa495cc752eeb7e096074b01d16d6605fb961f3ea3f4f3e3407ae2c4887d05cf
Jul 15 20:56:13.773941 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:13.773881    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bdvjc" event={"ID":"e5286d79-383d-4b78-a2b3-b8b6fea1b534","Type":"ContainerStarted","Data":"aa495cc752eeb7e096074b01d16d6605fb961f3ea3f4f3e3407ae2c4887d05cf"}
Jul 15 20:56:14.031202 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:14.031120    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:56:14.895481 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:14.895431    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:56:16.981434 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:16.981397    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:56:17.793761 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:17.793709    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bdvjc" event={"ID":"e5286d79-383d-4b78-a2b3-b8b6fea1b534","Type":"ContainerStarted","Data":"2275da0269596d6acb65fd48af2e2e9ed43f5a30854484f3b9a2f2abafcacc19"}
Jul 15 20:56:17.794110 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:17.793771    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bdvjc"
Jul 15 20:56:17.795391 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:17.795363    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t4rtd" event={"ID":"08bb52d6-c6ee-4072-985f-9891358602de","Type":"ContainerStarted","Data":"c6359790a06fee7138afdd8d94fd14a53d2d8a77b2fd98b8111c0238f2efe54b"}
Jul 15 20:56:17.795502 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:17.795491    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t4rtd"
Jul 15 20:56:17.817764 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:17.817720    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bdvjc" podStartSLOduration=2.416908444 podStartE2EDuration="5.817707416s" totalImagesPullingTime="3.400798972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:56:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:56:17.815933276 +0000 UTC m=+510.964971335" watchObservedRunningTime="2026-07-15 20:56:17.817707416 +0000 UTC m=+510.966745472"
Jul 15 20:56:17.845676 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:17.845630    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t4rtd" podStartSLOduration=1.625989649 podStartE2EDuration="7.845612497s" totalImagesPullingTime="6.219622848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:56:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:56:17.84328085 +0000 UTC m=+510.992318907" watchObservedRunningTime="2026-07-15 20:56:17.845612497 +0000 UTC m=+510.994650553"
Jul 15 20:56:18.252405 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:18.252361    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:56:18.484381 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:18.484343    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:56:19.819984 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:19.819942    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:56:19.910147 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:19.910119    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:56:21.651283 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:21.651245    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:56:21.731112 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:21.731073    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:56:28.809254 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:28.809223    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bdvjc"
Jul 15 20:56:28.809725 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:28.809289    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t4rtd"
Jul 15 20:56:30.333230 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:30.333190    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:56:30.469128 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:30.469090    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:56:30.651325 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:30.651240    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:56:30.986010 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:30.985979    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t4rtd"]
Jul 15 20:56:30.986232 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:30.986150    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t4rtd" podUID="08bb52d6-c6ee-4072-985f-9891358602de" containerName="manager" containerID="cri-o://c6359790a06fee7138afdd8d94fd14a53d2d8a77b2fd98b8111c0238f2efe54b" gracePeriod=2
Jul 15 20:56:31.000784 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:31.000753    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t4rtd"]
Jul 15 20:56:31.005591 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:56:31.005561    2566 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"08bb52d6-c6ee-4072-985f-9891358602de\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-t4rtd\": pods \"kuadrant-operator-controller-manager-8684c9ff49-t4rtd\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t4rtd"
Jul 15 20:56:31.017179 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:31.017150    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c24lt"]
Jul 15 20:56:31.017622 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:31.017605    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08bb52d6-c6ee-4072-985f-9891358602de" containerName="manager"
Jul 15 20:56:31.017668 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:31.017627    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="08bb52d6-c6ee-4072-985f-9891358602de" containerName="manager"
Jul 15 20:56:31.017714 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:31.017692    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="08bb52d6-c6ee-4072-985f-9891358602de" containerName="manager"
Jul 15 20:56:31.020559 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:31.020541    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c24lt"
Jul 15 20:56:31.023590 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:56:31.023557    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-t4rtd\" is forbidden: User \"system:node:ip-10-0-137-234.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-234.ec2.internal' and this object" podUID="08bb52d6-c6ee-4072-985f-9891358602de" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t4rtd"
Jul 15 20:56:31.043849 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:31.043819    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c24lt"]
Jul 15 20:56:31.155379 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:31.155345    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dwbl\" (UniqueName: \"kubernetes.io/projected/5ad16b5b-62b8-4133-948d-19e7b6374257-kube-api-access-4dwbl\") pod \"kuadrant-operator-controller-manager-8684c9ff49-c24lt\" (UID: \"5ad16b5b-62b8-4133-948d-19e7b6374257\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c24lt"
Jul 15 20:56:31.155543 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:31.155412    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5ad16b5b-62b8-4133-948d-19e7b6374257-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-c24lt\" (UID: \"5ad16b5b-62b8-4133-948d-19e7b6374257\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c24lt"
Jul 15 20:56:31.230222 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:31.230195    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t4rtd"
Jul 15 20:56:31.233340 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:56:31.233310    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-t4rtd\" is forbidden: User \"system:node:ip-10-0-137-234.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-234.ec2.internal' and this object" podUID="08bb52d6-c6ee-4072-985f-9891358602de" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t4rtd"
Jul 15 20:56:31.255966 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:31.255869    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dwbl\" (UniqueName: \"kubernetes.io/projected/5ad16b5b-62b8-4133-948d-19e7b6374257-kube-api-access-4dwbl\") pod \"kuadrant-operator-controller-manager-8684c9ff49-c24lt\" (UID: \"5ad16b5b-62b8-4133-948d-19e7b6374257\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c24lt"
Jul 15 20:56:31.256100 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:31.255970    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5ad16b5b-62b8-4133-948d-19e7b6374257-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-c24lt\" (UID: \"5ad16b5b-62b8-4133-948d-19e7b6374257\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c24lt"
Jul 15 20:56:31.256337 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:31.256317    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5ad16b5b-62b8-4133-948d-19e7b6374257-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-c24lt\" (UID: \"5ad16b5b-62b8-4133-948d-19e7b6374257\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c24lt"
Jul 15 20:56:31.270102 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:31.270077    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dwbl\" (UniqueName: \"kubernetes.io/projected/5ad16b5b-62b8-4133-948d-19e7b6374257-kube-api-access-4dwbl\") pod \"kuadrant-operator-controller-manager-8684c9ff49-c24lt\" (UID: \"5ad16b5b-62b8-4133-948d-19e7b6374257\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c24lt"
Jul 15 20:56:31.357137 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:31.357107    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/08bb52d6-c6ee-4072-985f-9891358602de-extensions-socket-volume\") pod \"08bb52d6-c6ee-4072-985f-9891358602de\" (UID: \"08bb52d6-c6ee-4072-985f-9891358602de\") "
Jul 15 20:56:31.357558 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:31.357156    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwz8t\" (UniqueName: \"kubernetes.io/projected/08bb52d6-c6ee-4072-985f-9891358602de-kube-api-access-jwz8t\") pod \"08bb52d6-c6ee-4072-985f-9891358602de\" (UID: \"08bb52d6-c6ee-4072-985f-9891358602de\") "
Jul 15 20:56:31.357692 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:31.357665    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08bb52d6-c6ee-4072-985f-9891358602de-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "08bb52d6-c6ee-4072-985f-9891358602de" (UID: "08bb52d6-c6ee-4072-985f-9891358602de"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 20:56:31.359324 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:31.359297    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08bb52d6-c6ee-4072-985f-9891358602de-kube-api-access-jwz8t" (OuterVolumeSpecName: "kube-api-access-jwz8t") pod "08bb52d6-c6ee-4072-985f-9891358602de" (UID: "08bb52d6-c6ee-4072-985f-9891358602de"). InnerVolumeSpecName "kube-api-access-jwz8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 20:56:31.376176 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:31.376155    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c24lt"
Jul 15 20:56:31.458460 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:31.458431    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwz8t\" (UniqueName: \"kubernetes.io/projected/08bb52d6-c6ee-4072-985f-9891358602de-kube-api-access-jwz8t\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 20:56:31.458460 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:31.458460    2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/08bb52d6-c6ee-4072-985f-9891358602de-extensions-socket-volume\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 20:56:31.517094 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:31.517068    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c24lt"]
Jul 15 20:56:31.519665 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:56:31.519625    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ad16b5b_62b8_4133_948d_19e7b6374257.slice/crio-142307dcd6cc3f09f693aae7abb3e446e154cae1ffffe0a6c0c8033068532d22 WatchSource:0}: Error finding container 142307dcd6cc3f09f693aae7abb3e446e154cae1ffffe0a6c0c8033068532d22: Status 404 returned error can't find the container with id 142307dcd6cc3f09f693aae7abb3e446e154cae1ffffe0a6c0c8033068532d22
Jul 15 20:56:31.755778 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:31.755749    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08bb52d6-c6ee-4072-985f-9891358602de" path="/var/lib/kubelet/pods/08bb52d6-c6ee-4072-985f-9891358602de/volumes"
Jul 15 20:56:31.862262 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:31.862174    2566 generic.go:356] "Generic (PLEG): container finished" podID="08bb52d6-c6ee-4072-985f-9891358602de" containerID="c6359790a06fee7138afdd8d94fd14a53d2d8a77b2fd98b8111c0238f2efe54b" exitCode=0
Jul 15 20:56:31.862262 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:31.862221    2566 scope.go:117] "RemoveContainer" containerID="c6359790a06fee7138afdd8d94fd14a53d2d8a77b2fd98b8111c0238f2efe54b"
Jul 15 20:56:31.862262 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:31.862246    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t4rtd"
Jul 15 20:56:31.864352 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:31.864263    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c24lt" event={"ID":"5ad16b5b-62b8-4133-948d-19e7b6374257","Type":"ContainerStarted","Data":"c1405ef848bd2bb3b0dea52aa7ce33aa64f6413c6e1332d635e52f87deed07c6"}
Jul 15 20:56:31.864352 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:31.864324    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c24lt" event={"ID":"5ad16b5b-62b8-4133-948d-19e7b6374257","Type":"ContainerStarted","Data":"142307dcd6cc3f09f693aae7abb3e446e154cae1ffffe0a6c0c8033068532d22"}
Jul 15 20:56:31.864763 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:31.864474    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c24lt"
Jul 15 20:56:31.870541 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:31.870524    2566 scope.go:117] "RemoveContainer" containerID="c6359790a06fee7138afdd8d94fd14a53d2d8a77b2fd98b8111c0238f2efe54b"
Jul 15 20:56:31.870799 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:56:31.870781    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6359790a06fee7138afdd8d94fd14a53d2d8a77b2fd98b8111c0238f2efe54b\": container with ID starting with c6359790a06fee7138afdd8d94fd14a53d2d8a77b2fd98b8111c0238f2efe54b not found: ID does not exist" containerID="c6359790a06fee7138afdd8d94fd14a53d2d8a77b2fd98b8111c0238f2efe54b"
Jul 15 20:56:31.870853 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:31.870808    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6359790a06fee7138afdd8d94fd14a53d2d8a77b2fd98b8111c0238f2efe54b"} err="failed to get container status \"c6359790a06fee7138afdd8d94fd14a53d2d8a77b2fd98b8111c0238f2efe54b\": rpc error: code = NotFound desc = could not find container \"c6359790a06fee7138afdd8d94fd14a53d2d8a77b2fd98b8111c0238f2efe54b\": container with ID starting with c6359790a06fee7138afdd8d94fd14a53d2d8a77b2fd98b8111c0238f2efe54b not found: ID does not exist"
Jul 15 20:56:32.420386 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:32.420330    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c24lt" podStartSLOduration=2.420310165 podStartE2EDuration="2.420310165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:56:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:56:31.886532011 +0000 UTC m=+525.035570080" watchObservedRunningTime="2026-07-15 20:56:32.420310165 +0000 UTC m=+525.569348221"
Jul 15 20:56:32.420970 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:32.420951    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5spsp"]
Jul 15 20:56:32.425299 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:32.425285    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5spsp"
Jul 15 20:56:32.438669 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:32.438647    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5spsp"]
Jul 15 20:56:32.568360 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:32.568317    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skbqc\" (UniqueName: \"kubernetes.io/projected/d1e08988-fd2c-4558-855e-cbdeb7d6f3f4-kube-api-access-skbqc\") pod \"kuadrant-operator-controller-manager-86878c769f-5spsp\" (UID: \"d1e08988-fd2c-4558-855e-cbdeb7d6f3f4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5spsp"
Jul 15 20:56:32.568536 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:32.568374    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d1e08988-fd2c-4558-855e-cbdeb7d6f3f4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-5spsp\" (UID: \"d1e08988-fd2c-4558-855e-cbdeb7d6f3f4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5spsp"
Jul 15 20:56:32.668812 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:32.668780    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skbqc\" (UniqueName: \"kubernetes.io/projected/d1e08988-fd2c-4558-855e-cbdeb7d6f3f4-kube-api-access-skbqc\") pod \"kuadrant-operator-controller-manager-86878c769f-5spsp\" (UID: \"d1e08988-fd2c-4558-855e-cbdeb7d6f3f4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5spsp"
Jul 15 20:56:32.669002 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:32.668819    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d1e08988-fd2c-4558-855e-cbdeb7d6f3f4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-5spsp\" (UID: \"d1e08988-fd2c-4558-855e-cbdeb7d6f3f4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5spsp"
Jul 15 20:56:32.669209 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:32.669192    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d1e08988-fd2c-4558-855e-cbdeb7d6f3f4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-5spsp\" (UID: \"d1e08988-fd2c-4558-855e-cbdeb7d6f3f4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5spsp"
Jul 15 20:56:32.686282 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:32.686217    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skbqc\" (UniqueName: \"kubernetes.io/projected/d1e08988-fd2c-4558-855e-cbdeb7d6f3f4-kube-api-access-skbqc\") pod \"kuadrant-operator-controller-manager-86878c769f-5spsp\" (UID: \"d1e08988-fd2c-4558-855e-cbdeb7d6f3f4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5spsp"
Jul 15 20:56:32.735282 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:32.735255    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5spsp"
Jul 15 20:56:32.756569 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:32.756540    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:56:32.857945 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:32.857890    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:56:32.872145 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:32.872126    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5spsp"]
Jul 15 20:56:32.874852 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:56:32.874832    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1e08988_fd2c_4558_855e_cbdeb7d6f3f4.slice/crio-819f07944f6561ef527ebbc39447e0c7f04c9486e9c7d1d119906cbbae95c03c WatchSource:0}: Error finding container 819f07944f6561ef527ebbc39447e0c7f04c9486e9c7d1d119906cbbae95c03c: Status 404 returned error can't find the container with id 819f07944f6561ef527ebbc39447e0c7f04c9486e9c7d1d119906cbbae95c03c
Jul 15 20:56:33.874866 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:33.874831    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5spsp" event={"ID":"d1e08988-fd2c-4558-855e-cbdeb7d6f3f4","Type":"ContainerStarted","Data":"1e23b67e08637289fcd329fabbfeec503152ac14fab2706251997a426053775c"}
Jul 15 20:56:33.875354 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:33.874887    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5spsp" event={"ID":"d1e08988-fd2c-4558-855e-cbdeb7d6f3f4","Type":"ContainerStarted","Data":"819f07944f6561ef527ebbc39447e0c7f04c9486e9c7d1d119906cbbae95c03c"}
Jul 15 20:56:33.875354 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:33.874940    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5spsp"
Jul 15 20:56:33.903221 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:33.903167    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5spsp" podStartSLOduration=1.903152445 podStartE2EDuration="1.903152445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:56:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:56:33.90172714 +0000 UTC m=+527.050765198" watchObservedRunningTime="2026-07-15 20:56:33.903152445 +0000 UTC m=+527.052190502"
Jul 15 20:56:37.180478 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:37.180439    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:56:37.334357 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:37.334313    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:56:39.315360 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:39.315275    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:56:39.415146 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:39.415115    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:56:42.871483 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:42.871457    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c24lt"
Jul 15 20:56:44.882018 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:44.881990    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5spsp"
Jul 15 20:56:44.936888 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:44.936856    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c24lt"]
Jul 15 20:56:44.937088 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:44.937064    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c24lt" podUID="5ad16b5b-62b8-4133-948d-19e7b6374257" containerName="manager" containerID="cri-o://c1405ef848bd2bb3b0dea52aa7ce33aa64f6413c6e1332d635e52f87deed07c6" gracePeriod=10
Jul 15 20:56:45.190583 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:45.190556    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c24lt"
Jul 15 20:56:45.269371 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:45.269344    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5ad16b5b-62b8-4133-948d-19e7b6374257-extensions-socket-volume\") pod \"5ad16b5b-62b8-4133-948d-19e7b6374257\" (UID: \"5ad16b5b-62b8-4133-948d-19e7b6374257\") "
Jul 15 20:56:45.269371 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:45.269379    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4dwbl\" (UniqueName: \"kubernetes.io/projected/5ad16b5b-62b8-4133-948d-19e7b6374257-kube-api-access-4dwbl\") pod \"5ad16b5b-62b8-4133-948d-19e7b6374257\" (UID: \"5ad16b5b-62b8-4133-948d-19e7b6374257\") "
Jul 15 20:56:45.269764 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:45.269737    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ad16b5b-62b8-4133-948d-19e7b6374257-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5ad16b5b-62b8-4133-948d-19e7b6374257" (UID: "5ad16b5b-62b8-4133-948d-19e7b6374257"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 20:56:45.271397 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:45.271377    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ad16b5b-62b8-4133-948d-19e7b6374257-kube-api-access-4dwbl" (OuterVolumeSpecName: "kube-api-access-4dwbl") pod "5ad16b5b-62b8-4133-948d-19e7b6374257" (UID: "5ad16b5b-62b8-4133-948d-19e7b6374257"). InnerVolumeSpecName "kube-api-access-4dwbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 20:56:45.369857 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:45.369815    2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5ad16b5b-62b8-4133-948d-19e7b6374257-extensions-socket-volume\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 20:56:45.369857 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:45.369851    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dwbl\" (UniqueName: \"kubernetes.io/projected/5ad16b5b-62b8-4133-948d-19e7b6374257-kube-api-access-4dwbl\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 20:56:45.925043 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:45.925012    2566 generic.go:356] "Generic (PLEG): container finished" podID="5ad16b5b-62b8-4133-948d-19e7b6374257" containerID="c1405ef848bd2bb3b0dea52aa7ce33aa64f6413c6e1332d635e52f87deed07c6" exitCode=0
Jul 15 20:56:45.925469 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:45.925060    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c24lt"
Jul 15 20:56:45.925469 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:45.925076    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c24lt" event={"ID":"5ad16b5b-62b8-4133-948d-19e7b6374257","Type":"ContainerDied","Data":"c1405ef848bd2bb3b0dea52aa7ce33aa64f6413c6e1332d635e52f87deed07c6"}
Jul 15 20:56:45.925469 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:45.925118    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c24lt" event={"ID":"5ad16b5b-62b8-4133-948d-19e7b6374257","Type":"ContainerDied","Data":"142307dcd6cc3f09f693aae7abb3e446e154cae1ffffe0a6c0c8033068532d22"}
Jul 15 20:56:45.925469 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:45.925140    2566 scope.go:117] "RemoveContainer" containerID="c1405ef848bd2bb3b0dea52aa7ce33aa64f6413c6e1332d635e52f87deed07c6"
Jul 15 20:56:45.933370 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:45.933352    2566 scope.go:117] "RemoveContainer" containerID="c1405ef848bd2bb3b0dea52aa7ce33aa64f6413c6e1332d635e52f87deed07c6"
Jul 15 20:56:45.933611 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:56:45.933592    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1405ef848bd2bb3b0dea52aa7ce33aa64f6413c6e1332d635e52f87deed07c6\": container with ID starting with c1405ef848bd2bb3b0dea52aa7ce33aa64f6413c6e1332d635e52f87deed07c6 not found: ID does not exist" containerID="c1405ef848bd2bb3b0dea52aa7ce33aa64f6413c6e1332d635e52f87deed07c6"
Jul 15 20:56:45.933675 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:45.933624    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1405ef848bd2bb3b0dea52aa7ce33aa64f6413c6e1332d635e52f87deed07c6"} err="failed to get container status \"c1405ef848bd2bb3b0dea52aa7ce33aa64f6413c6e1332d635e52f87deed07c6\": rpc error: code = NotFound desc = could not find container \"c1405ef848bd2bb3b0dea52aa7ce33aa64f6413c6e1332d635e52f87deed07c6\": container with ID starting with c1405ef848bd2bb3b0dea52aa7ce33aa64f6413c6e1332d635e52f87deed07c6 not found: ID does not exist"
Jul 15 20:56:45.947521 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:45.947500    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c24lt"]
Jul 15 20:56:45.951669 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:45.951648    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c24lt"]
Jul 15 20:56:47.756565 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:47.756532    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ad16b5b-62b8-4133-948d-19e7b6374257" path="/var/lib/kubelet/pods/5ad16b5b-62b8-4133-948d-19e7b6374257/volumes"
Jul 15 20:56:48.222438 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:48.222399    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:56:48.322837 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:48.322807    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:56:48.424305 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:48.424273    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:56:50.644057 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:50.644022    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:56:50.746972 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:50.746926    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:56:55.030032 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:55.030000    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:56:55.155564 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:55.155533    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:56:57.998906 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:57.998856    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:56:58.075423 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:56:58.075396    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:57:01.388037 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:01.388001    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:57:01.526064 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:01.526028    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:57:01.890757 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:01.890723    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:57:01.990889 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:01.990851    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:57:03.829286 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:03.829254    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:57:03.900719 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:03.900679    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:57:03.999106 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:03.999074    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:57:04.625186 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:04.625152    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:57:04.699117 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:04.699084    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:57:06.483203 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:06.483174    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:57:06.585428 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:06.585391    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:57:08.939116 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:08.939078    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:57:09.015031 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:09.015000    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:57:25.108995 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:25.108960    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:57:25.201937 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:25.201877    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:57:25.666005 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:25.665968    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:57:29.027711 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:29.027682    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:57:34.634504 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:34.634466    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:57:38.253366 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:38.253310    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:57:38.681544 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:38.681457    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:57:39.826019 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:39.825984    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:57:45.601713 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:45.601675    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:57:45.624352 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:45.624315    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7f6df96b69-gf87s"]
Jul 15 20:57:45.624787 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:45.624767    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ad16b5b-62b8-4133-948d-19e7b6374257" containerName="manager"
Jul 15 20:57:45.624853 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:45.624792    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ad16b5b-62b8-4133-948d-19e7b6374257" containerName="manager"
Jul 15 20:57:45.624936 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:45.624874    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ad16b5b-62b8-4133-948d-19e7b6374257" containerName="manager"
Jul 15 20:57:45.627056 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:45.627039    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f6df96b69-gf87s"
Jul 15 20:57:45.630105 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:45.630088    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 15 20:57:45.630207 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:45.630107    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-s478v\""
Jul 15 20:57:45.630207 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:45.630092    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 15 20:57:45.636345 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:45.636320    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f6df96b69-gf87s"]
Jul 15 20:57:45.697941 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:45.697885    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/867d81f0-85e5-4a99-ab1a-3fcf588e0c37-webhook-cert\") pod \"maas-controller-7f6df96b69-gf87s\" (UID: \"867d81f0-85e5-4a99-ab1a-3fcf588e0c37\") " pod="opendatahub/maas-controller-7f6df96b69-gf87s"
Jul 15 20:57:45.698115 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:45.697952    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc29s\" (UniqueName: \"kubernetes.io/projected/867d81f0-85e5-4a99-ab1a-3fcf588e0c37-kube-api-access-fc29s\") pod \"maas-controller-7f6df96b69-gf87s\" (UID: \"867d81f0-85e5-4a99-ab1a-3fcf588e0c37\") " pod="opendatahub/maas-controller-7f6df96b69-gf87s"
Jul 15 20:57:45.701350 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:45.701324    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:57:45.799443 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:45.799410    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/867d81f0-85e5-4a99-ab1a-3fcf588e0c37-webhook-cert\") pod \"maas-controller-7f6df96b69-gf87s\" (UID: \"867d81f0-85e5-4a99-ab1a-3fcf588e0c37\") " pod="opendatahub/maas-controller-7f6df96b69-gf87s"
Jul 15 20:57:45.799631 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:57:45.799539    2566 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Jul 15 20:57:45.799631 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:57:45.799615    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/867d81f0-85e5-4a99-ab1a-3fcf588e0c37-webhook-cert podName:867d81f0-85e5-4a99-ab1a-3fcf588e0c37 nodeName:}" failed. No retries permitted until 2026-07-15 20:57:46.299593146 +0000 UTC m=+599.448631191 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/867d81f0-85e5-4a99-ab1a-3fcf588e0c37-webhook-cert") pod "maas-controller-7f6df96b69-gf87s" (UID: "867d81f0-85e5-4a99-ab1a-3fcf588e0c37") : secret "maas-controller-webhook-cert" not found
Jul 15 20:57:45.799774 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:45.799653    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc29s\" (UniqueName: \"kubernetes.io/projected/867d81f0-85e5-4a99-ab1a-3fcf588e0c37-kube-api-access-fc29s\") pod \"maas-controller-7f6df96b69-gf87s\" (UID: \"867d81f0-85e5-4a99-ab1a-3fcf588e0c37\") " pod="opendatahub/maas-controller-7f6df96b69-gf87s"
Jul 15 20:57:45.803368 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:45.803338    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:57:45.810736 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:45.810704    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc29s\" (UniqueName: \"kubernetes.io/projected/867d81f0-85e5-4a99-ab1a-3fcf588e0c37-kube-api-access-fc29s\") pod \"maas-controller-7f6df96b69-gf87s\" (UID: \"867d81f0-85e5-4a99-ab1a-3fcf588e0c37\") " pod="opendatahub/maas-controller-7f6df96b69-gf87s"
Jul 15 20:57:45.820844 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:45.820821    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-567446f9b-8wqwj"]
Jul 15 20:57:45.823294 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:45.823272    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-567446f9b-8wqwj"
Jul 15 20:57:45.835512 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:45.835486    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-567446f9b-8wqwj"]
Jul 15 20:57:45.901493 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:45.901403    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e9593db8-063d-4d3e-8420-8df1a687f3d5-webhook-cert\") pod \"maas-controller-567446f9b-8wqwj\" (UID: \"e9593db8-063d-4d3e-8420-8df1a687f3d5\") " pod="opendatahub/maas-controller-567446f9b-8wqwj"
Jul 15 20:57:45.901493 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:45.901480    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc4lz\" (UniqueName: \"kubernetes.io/projected/e9593db8-063d-4d3e-8420-8df1a687f3d5-kube-api-access-nc4lz\") pod \"maas-controller-567446f9b-8wqwj\" (UID: \"e9593db8-063d-4d3e-8420-8df1a687f3d5\") " pod="opendatahub/maas-controller-567446f9b-8wqwj"
Jul 15 20:57:46.002167 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:46.002124    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc4lz\" (UniqueName: \"kubernetes.io/projected/e9593db8-063d-4d3e-8420-8df1a687f3d5-kube-api-access-nc4lz\") pod \"maas-controller-567446f9b-8wqwj\" (UID: \"e9593db8-063d-4d3e-8420-8df1a687f3d5\") " pod="opendatahub/maas-controller-567446f9b-8wqwj"
Jul 15 20:57:46.002372 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:46.002190    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e9593db8-063d-4d3e-8420-8df1a687f3d5-webhook-cert\") pod \"maas-controller-567446f9b-8wqwj\" (UID: \"e9593db8-063d-4d3e-8420-8df1a687f3d5\") " pod="opendatahub/maas-controller-567446f9b-8wqwj"
Jul 15 20:57:46.002372 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:57:46.002289    2566 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Jul 15 20:57:46.002372 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:57:46.002363    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9593db8-063d-4d3e-8420-8df1a687f3d5-webhook-cert podName:e9593db8-063d-4d3e-8420-8df1a687f3d5 nodeName:}" failed. No retries permitted until 2026-07-15 20:57:46.502347847 +0000 UTC m=+599.651385882 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/e9593db8-063d-4d3e-8420-8df1a687f3d5-webhook-cert") pod "maas-controller-567446f9b-8wqwj" (UID: "e9593db8-063d-4d3e-8420-8df1a687f3d5") : secret "maas-controller-webhook-cert" not found
Jul 15 20:57:46.011966 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:46.011930    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc4lz\" (UniqueName: \"kubernetes.io/projected/e9593db8-063d-4d3e-8420-8df1a687f3d5-kube-api-access-nc4lz\") pod \"maas-controller-567446f9b-8wqwj\" (UID: \"e9593db8-063d-4d3e-8420-8df1a687f3d5\") " pod="opendatahub/maas-controller-567446f9b-8wqwj"
Jul 15 20:57:46.141437 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:46.141401    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:57:46.304871 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:46.304836    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/867d81f0-85e5-4a99-ab1a-3fcf588e0c37-webhook-cert\") pod \"maas-controller-7f6df96b69-gf87s\" (UID: \"867d81f0-85e5-4a99-ab1a-3fcf588e0c37\") " pod="opendatahub/maas-controller-7f6df96b69-gf87s"
Jul 15 20:57:46.307140 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:46.307122    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/867d81f0-85e5-4a99-ab1a-3fcf588e0c37-webhook-cert\") pod \"maas-controller-7f6df96b69-gf87s\" (UID: \"867d81f0-85e5-4a99-ab1a-3fcf588e0c37\") " pod="opendatahub/maas-controller-7f6df96b69-gf87s"
Jul 15 20:57:46.506998 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:46.506957    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e9593db8-063d-4d3e-8420-8df1a687f3d5-webhook-cert\") pod \"maas-controller-567446f9b-8wqwj\" (UID: \"e9593db8-063d-4d3e-8420-8df1a687f3d5\") " pod="opendatahub/maas-controller-567446f9b-8wqwj"
Jul 15 20:57:46.509360 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:46.509342    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e9593db8-063d-4d3e-8420-8df1a687f3d5-webhook-cert\") pod \"maas-controller-567446f9b-8wqwj\" (UID: \"e9593db8-063d-4d3e-8420-8df1a687f3d5\") " pod="opendatahub/maas-controller-567446f9b-8wqwj"
Jul 15 20:57:46.537947 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:46.537920    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f6df96b69-gf87s"
Jul 15 20:57:46.668927 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:46.668823    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f6df96b69-gf87s"]
Jul 15 20:57:46.671222 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:57:46.671198    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod867d81f0_85e5_4a99_ab1a_3fcf588e0c37.slice/crio-0389ccd6b8042b668c6ef5d23db164bf8db67b569d945e666623edbd29675592 WatchSource:0}: Error finding container 0389ccd6b8042b668c6ef5d23db164bf8db67b569d945e666623edbd29675592: Status 404 returned error can't find the container with id 0389ccd6b8042b668c6ef5d23db164bf8db67b569d945e666623edbd29675592
Jul 15 20:57:46.735373 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:46.735336    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-567446f9b-8wqwj"
Jul 15 20:57:46.868066 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:46.868042    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-567446f9b-8wqwj"]
Jul 15 20:57:46.870039 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:57:46.870016    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9593db8_063d_4d3e_8420_8df1a687f3d5.slice/crio-99ebe426e3103f2f4067039a5e70f6b4f1db8cc3255de27a3437a308386b1ed0 WatchSource:0}: Error finding container 99ebe426e3103f2f4067039a5e70f6b4f1db8cc3255de27a3437a308386b1ed0: Status 404 returned error can't find the container with id 99ebe426e3103f2f4067039a5e70f6b4f1db8cc3255de27a3437a308386b1ed0
Jul 15 20:57:47.175158 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:47.175062    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-567446f9b-8wqwj" event={"ID":"e9593db8-063d-4d3e-8420-8df1a687f3d5","Type":"ContainerStarted","Data":"99ebe426e3103f2f4067039a5e70f6b4f1db8cc3255de27a3437a308386b1ed0"}
Jul 15 20:57:47.176313 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:47.176285    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f6df96b69-gf87s" event={"ID":"867d81f0-85e5-4a99-ab1a-3fcf588e0c37","Type":"ContainerStarted","Data":"0389ccd6b8042b668c6ef5d23db164bf8db67b569d945e666623edbd29675592"}
Jul 15 20:57:47.648791 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:47.648747    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-652lk_f9a159bb-39c4-4cca-92ff-1c74d739e311/ovn-acl-logging/0.log"
Jul 15 20:57:47.650830 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:47.650800    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-652lk_f9a159bb-39c4-4cca-92ff-1c74d739e311/ovn-acl-logging/0.log"
Jul 15 20:57:49.307498 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:49.307474    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 15 20:57:49.522272 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:49.522236    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:57:49.623680 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:49.623592    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:57:50.201306 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:50.201271    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-567446f9b-8wqwj" event={"ID":"e9593db8-063d-4d3e-8420-8df1a687f3d5","Type":"ContainerStarted","Data":"86efb21a65b88324da0ef11f1eeb2e1c6b1919d5d40e1ea24e55064a34a18019"}
Jul 15 20:57:50.201492 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:50.201434    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-567446f9b-8wqwj"
Jul 15 20:57:50.202876 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:50.202847    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f6df96b69-gf87s" event={"ID":"867d81f0-85e5-4a99-ab1a-3fcf588e0c37","Type":"ContainerStarted","Data":"ab1c36e5dc4ad8143e5af625f27813dd55222e6f70e67ba0dfae4ea26adaad3c"}
Jul 15 20:57:50.203029 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:50.203000    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7f6df96b69-gf87s"
Jul 15 20:57:50.224198 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:50.224151    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-567446f9b-8wqwj" podStartSLOduration=2.784152638 podStartE2EDuration="5.224138061s" totalImagesPullingTime="2.439985423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:57:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:57:50.222211072 +0000 UTC m=+603.371249166" watchObservedRunningTime="2026-07-15 20:57:50.224138061 +0000 UTC m=+603.373176118"
Jul 15 20:57:50.245088 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:50.245043    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7f6df96b69-gf87s" podStartSLOduration=2.6132590479999998 podStartE2EDuration="5.245026655s" totalImagesPullingTime="2.631767607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:57:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:57:50.244306236 +0000 UTC m=+603.393344292" watchObservedRunningTime="2026-07-15 20:57:50.245026655 +0000 UTC m=+603.394064712"
Jul 15 20:57:50.743297 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:50.743259    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:57:50.841587 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:50.841548    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:57:58.380346 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:57:58.380314    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:58:01.209908 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:01.209872    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7f6df96b69-gf87s"
Jul 15 20:58:01.210326 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:01.209974    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-567446f9b-8wqwj"
Jul 15 20:58:01.299256 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:01.299226    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7f6df96b69-gf87s"]
Jul 15 20:58:01.299438 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:01.299416    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7f6df96b69-gf87s" podUID="867d81f0-85e5-4a99-ab1a-3fcf588e0c37" containerName="manager" containerID="cri-o://ab1c36e5dc4ad8143e5af625f27813dd55222e6f70e67ba0dfae4ea26adaad3c" gracePeriod=30
Jul 15 20:58:01.539814 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:01.539788    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f6df96b69-gf87s"
Jul 15 20:58:01.635467 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:01.635434    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fc29s\" (UniqueName: \"kubernetes.io/projected/867d81f0-85e5-4a99-ab1a-3fcf588e0c37-kube-api-access-fc29s\") pod \"867d81f0-85e5-4a99-ab1a-3fcf588e0c37\" (UID: \"867d81f0-85e5-4a99-ab1a-3fcf588e0c37\") "
Jul 15 20:58:01.635626 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:01.635531    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/867d81f0-85e5-4a99-ab1a-3fcf588e0c37-webhook-cert\") pod \"867d81f0-85e5-4a99-ab1a-3fcf588e0c37\" (UID: \"867d81f0-85e5-4a99-ab1a-3fcf588e0c37\") "
Jul 15 20:58:01.637761 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:01.637727    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/867d81f0-85e5-4a99-ab1a-3fcf588e0c37-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "867d81f0-85e5-4a99-ab1a-3fcf588e0c37" (UID: "867d81f0-85e5-4a99-ab1a-3fcf588e0c37"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 20:58:01.637761 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:01.637729    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/867d81f0-85e5-4a99-ab1a-3fcf588e0c37-kube-api-access-fc29s" (OuterVolumeSpecName: "kube-api-access-fc29s") pod "867d81f0-85e5-4a99-ab1a-3fcf588e0c37" (UID: "867d81f0-85e5-4a99-ab1a-3fcf588e0c37"). InnerVolumeSpecName "kube-api-access-fc29s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 20:58:01.736580 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:01.736487    2566 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/867d81f0-85e5-4a99-ab1a-3fcf588e0c37-webhook-cert\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 20:58:01.736580 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:01.736517    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fc29s\" (UniqueName: \"kubernetes.io/projected/867d81f0-85e5-4a99-ab1a-3fcf588e0c37-kube-api-access-fc29s\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 20:58:02.252545 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:02.252505    2566 generic.go:356] "Generic (PLEG): container finished" podID="867d81f0-85e5-4a99-ab1a-3fcf588e0c37" containerID="ab1c36e5dc4ad8143e5af625f27813dd55222e6f70e67ba0dfae4ea26adaad3c" exitCode=0
Jul 15 20:58:02.252995 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:02.252552    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f6df96b69-gf87s" event={"ID":"867d81f0-85e5-4a99-ab1a-3fcf588e0c37","Type":"ContainerDied","Data":"ab1c36e5dc4ad8143e5af625f27813dd55222e6f70e67ba0dfae4ea26adaad3c"}
Jul 15 20:58:02.252995 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:02.252571    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f6df96b69-gf87s"
Jul 15 20:58:02.252995 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:02.252601    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f6df96b69-gf87s" event={"ID":"867d81f0-85e5-4a99-ab1a-3fcf588e0c37","Type":"ContainerDied","Data":"0389ccd6b8042b668c6ef5d23db164bf8db67b569d945e666623edbd29675592"}
Jul 15 20:58:02.252995 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:02.252623    2566 scope.go:117] "RemoveContainer" containerID="ab1c36e5dc4ad8143e5af625f27813dd55222e6f70e67ba0dfae4ea26adaad3c"
Jul 15 20:58:02.260810 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:02.260789    2566 scope.go:117] "RemoveContainer" containerID="ab1c36e5dc4ad8143e5af625f27813dd55222e6f70e67ba0dfae4ea26adaad3c"
Jul 15 20:58:02.261097 ip-10-0-137-234 kubenswrapper[2566]: E0715 20:58:02.261078    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab1c36e5dc4ad8143e5af625f27813dd55222e6f70e67ba0dfae4ea26adaad3c\": container with ID starting with ab1c36e5dc4ad8143e5af625f27813dd55222e6f70e67ba0dfae4ea26adaad3c not found: ID does not exist" containerID="ab1c36e5dc4ad8143e5af625f27813dd55222e6f70e67ba0dfae4ea26adaad3c"
Jul 15 20:58:02.261155 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:02.261106    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab1c36e5dc4ad8143e5af625f27813dd55222e6f70e67ba0dfae4ea26adaad3c"} err="failed to get container status \"ab1c36e5dc4ad8143e5af625f27813dd55222e6f70e67ba0dfae4ea26adaad3c\": rpc error: code = NotFound desc = could not find container \"ab1c36e5dc4ad8143e5af625f27813dd55222e6f70e67ba0dfae4ea26adaad3c\": container with ID starting with ab1c36e5dc4ad8143e5af625f27813dd55222e6f70e67ba0dfae4ea26adaad3c not found: ID does not exist"
Jul 15 20:58:02.270478 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:02.270454    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7f6df96b69-gf87s"]
Jul 15 20:58:02.274450 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:02.274429    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7f6df96b69-gf87s"]
Jul 15 20:58:03.756581 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:03.756549    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="867d81f0-85e5-4a99-ab1a-3fcf588e0c37" path="/var/lib/kubelet/pods/867d81f0-85e5-4a99-ab1a-3fcf588e0c37/volumes"
Jul 15 20:58:20.885869 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:20.885777    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:58:22.324387 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:22.324346    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:58:22.920626 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:22.920595    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:58:23.044676 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:23.044641    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:58:23.112123 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:23.112085    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-thxk8"]
Jul 15 20:58:23.112411 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:23.112398    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="867d81f0-85e5-4a99-ab1a-3fcf588e0c37" containerName="manager"
Jul 15 20:58:23.112460 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:23.112413    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="867d81f0-85e5-4a99-ab1a-3fcf588e0c37" containerName="manager"
Jul 15 20:58:23.112491 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:23.112469    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="867d81f0-85e5-4a99-ab1a-3fcf588e0c37" containerName="manager"
Jul 15 20:58:23.117400 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:23.117382    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-thxk8"
Jul 15 20:58:23.121149 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:23.121127    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 15 20:58:23.121348 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:23.121330    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 15 20:58:23.121579 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:23.121562    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 15 20:58:23.142866 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:23.142839    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-thxk8"]
Jul 15 20:58:23.203371 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:23.203335    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce43b1a9-5e27-4ac1-8430-644dd0d9b106-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-thxk8\" (UID: \"ce43b1a9-5e27-4ac1-8430-644dd0d9b106\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-thxk8"
Jul 15 20:58:23.203548 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:23.203373    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zt6x\" (UniqueName: \"kubernetes.io/projected/ce43b1a9-5e27-4ac1-8430-644dd0d9b106-kube-api-access-6zt6x\") pod \"payload-pre-processing-74584bb9f8-thxk8\" (UID: \"ce43b1a9-5e27-4ac1-8430-644dd0d9b106\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-thxk8"
Jul 15 20:58:23.304264 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:23.304225    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce43b1a9-5e27-4ac1-8430-644dd0d9b106-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-thxk8\" (UID: \"ce43b1a9-5e27-4ac1-8430-644dd0d9b106\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-thxk8"
Jul 15 20:58:23.304264 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:23.304266    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zt6x\" (UniqueName: \"kubernetes.io/projected/ce43b1a9-5e27-4ac1-8430-644dd0d9b106-kube-api-access-6zt6x\") pod \"payload-pre-processing-74584bb9f8-thxk8\" (UID: \"ce43b1a9-5e27-4ac1-8430-644dd0d9b106\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-thxk8"
Jul 15 20:58:23.304945 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:23.304925    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce43b1a9-5e27-4ac1-8430-644dd0d9b106-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-thxk8\" (UID: \"ce43b1a9-5e27-4ac1-8430-644dd0d9b106\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-thxk8"
Jul 15 20:58:23.316712 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:23.316675    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zt6x\" (UniqueName: \"kubernetes.io/projected/ce43b1a9-5e27-4ac1-8430-644dd0d9b106-kube-api-access-6zt6x\") pod \"payload-pre-processing-74584bb9f8-thxk8\" (UID: \"ce43b1a9-5e27-4ac1-8430-644dd0d9b106\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-thxk8"
Jul 15 20:58:23.427711 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:23.427646    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-thxk8"
Jul 15 20:58:23.575723 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:23.575692    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-thxk8"]
Jul 15 20:58:23.577743 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:58:23.577716    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce43b1a9_5e27_4ac1_8430_644dd0d9b106.slice/crio-5445ca360633110d0a339237ae5cc16c990120deca60f9b19b69ec170538f478 WatchSource:0}: Error finding container 5445ca360633110d0a339237ae5cc16c990120deca60f9b19b69ec170538f478: Status 404 returned error can't find the container with id 5445ca360633110d0a339237ae5cc16c990120deca60f9b19b69ec170538f478
Jul 15 20:58:24.355341 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:24.355301    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-thxk8" event={"ID":"ce43b1a9-5e27-4ac1-8430-644dd0d9b106","Type":"ContainerStarted","Data":"5445ca360633110d0a339237ae5cc16c990120deca60f9b19b69ec170538f478"}
Jul 15 20:58:25.361877 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:25.361839    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-thxk8" event={"ID":"ce43b1a9-5e27-4ac1-8430-644dd0d9b106","Type":"ContainerStarted","Data":"6e3fe779bff60004a40849d051f2c411b8a8e923a0121eb1fceb4837f24ec5b5"}
Jul 15 20:58:25.362258 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:25.361914    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-thxk8"
Jul 15 20:58:25.387115 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:25.387066    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-74584bb9f8-thxk8" podStartSLOduration=1.197837592 podStartE2EDuration="2.387047601s" totalImagesPullingTime="1.189210009s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:58:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:58:25.386687572 +0000 UTC m=+638.535725638" watchObservedRunningTime="2026-07-15 20:58:25.387047601 +0000 UTC m=+638.536085658"
Jul 15 20:58:33.277746 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:33.277705    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:58:33.382035 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:33.381999    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:58:36.369001 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:36.368967    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-thxk8"
Jul 15 20:58:46.227042 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:46.227007    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zb9gv"]
Jul 15 20:58:46.230816 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:46.230790    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zb9gv"
Jul 15 20:58:46.234111 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:46.234086    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-4f6jp\""
Jul 15 20:58:46.235341 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:46.235297    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 15 20:58:46.235449 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:46.235297    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 15 20:58:46.235519 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:46.235500    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 15 20:58:46.241802 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:46.241775    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zb9gv"]
Jul 15 20:58:46.272173 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:46.272129    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:58:46.305638 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:46.305601    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0d97cd6-8f3d-4713-80e2-43d8be42fc6c-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zb9gv\" (UID: \"b0d97cd6-8f3d-4713-80e2-43d8be42fc6c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zb9gv"
Jul 15 20:58:46.305638 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:46.305645    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0d97cd6-8f3d-4713-80e2-43d8be42fc6c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zb9gv\" (UID: \"b0d97cd6-8f3d-4713-80e2-43d8be42fc6c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zb9gv"
Jul 15 20:58:46.305833 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:46.305663    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0d97cd6-8f3d-4713-80e2-43d8be42fc6c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zb9gv\" (UID: \"b0d97cd6-8f3d-4713-80e2-43d8be42fc6c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zb9gv"
Jul 15 20:58:46.305833 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:46.305747    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0d97cd6-8f3d-4713-80e2-43d8be42fc6c-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zb9gv\" (UID: \"b0d97cd6-8f3d-4713-80e2-43d8be42fc6c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zb9gv"
Jul 15 20:58:46.305833 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:46.305774    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0d97cd6-8f3d-4713-80e2-43d8be42fc6c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zb9gv\" (UID: \"b0d97cd6-8f3d-4713-80e2-43d8be42fc6c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zb9gv"
Jul 15 20:58:46.305833 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:46.305800    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njjmg\" (UniqueName: \"kubernetes.io/projected/b0d97cd6-8f3d-4713-80e2-43d8be42fc6c-kube-api-access-njjmg\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zb9gv\" (UID: \"b0d97cd6-8f3d-4713-80e2-43d8be42fc6c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zb9gv"
Jul 15 20:58:46.406687 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:46.406641    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0d97cd6-8f3d-4713-80e2-43d8be42fc6c-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zb9gv\" (UID: \"b0d97cd6-8f3d-4713-80e2-43d8be42fc6c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zb9gv"
Jul 15 20:58:46.406687 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:46.406684    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0d97cd6-8f3d-4713-80e2-43d8be42fc6c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zb9gv\" (UID: \"b0d97cd6-8f3d-4713-80e2-43d8be42fc6c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zb9gv"
Jul 15 20:58:46.406949 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:46.406712    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njjmg\" (UniqueName: \"kubernetes.io/projected/b0d97cd6-8f3d-4713-80e2-43d8be42fc6c-kube-api-access-njjmg\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zb9gv\" (UID: \"b0d97cd6-8f3d-4713-80e2-43d8be42fc6c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zb9gv"
Jul 15 20:58:46.406949 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:46.406799    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0d97cd6-8f3d-4713-80e2-43d8be42fc6c-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zb9gv\" (UID: \"b0d97cd6-8f3d-4713-80e2-43d8be42fc6c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zb9gv"
Jul 15 20:58:46.406949 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:46.406838    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0d97cd6-8f3d-4713-80e2-43d8be42fc6c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zb9gv\" (UID: \"b0d97cd6-8f3d-4713-80e2-43d8be42fc6c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zb9gv"
Jul 15 20:58:46.406949 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:46.406863    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0d97cd6-8f3d-4713-80e2-43d8be42fc6c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zb9gv\" (UID: \"b0d97cd6-8f3d-4713-80e2-43d8be42fc6c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zb9gv"
Jul 15 20:58:46.407280 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:46.407253    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0d97cd6-8f3d-4713-80e2-43d8be42fc6c-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zb9gv\" (UID: \"b0d97cd6-8f3d-4713-80e2-43d8be42fc6c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zb9gv"
Jul 15 20:58:46.407280 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:46.407268    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0d97cd6-8f3d-4713-80e2-43d8be42fc6c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zb9gv\" (UID: \"b0d97cd6-8f3d-4713-80e2-43d8be42fc6c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zb9gv"
Jul 15 20:58:46.407476 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:46.407328    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0d97cd6-8f3d-4713-80e2-43d8be42fc6c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zb9gv\" (UID: \"b0d97cd6-8f3d-4713-80e2-43d8be42fc6c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zb9gv"
Jul 15 20:58:46.409082 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:46.409059    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0d97cd6-8f3d-4713-80e2-43d8be42fc6c-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zb9gv\" (UID: \"b0d97cd6-8f3d-4713-80e2-43d8be42fc6c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zb9gv"
Jul 15 20:58:46.409322 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:46.409303    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0d97cd6-8f3d-4713-80e2-43d8be42fc6c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zb9gv\" (UID: \"b0d97cd6-8f3d-4713-80e2-43d8be42fc6c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zb9gv"
Jul 15 20:58:46.416478 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:46.416452    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njjmg\" (UniqueName: \"kubernetes.io/projected/b0d97cd6-8f3d-4713-80e2-43d8be42fc6c-kube-api-access-njjmg\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zb9gv\" (UID: \"b0d97cd6-8f3d-4713-80e2-43d8be42fc6c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zb9gv"
Jul 15 20:58:46.546750 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:46.546663    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zb9gv"
Jul 15 20:58:46.683002 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:46.682968    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zb9gv"]
Jul 15 20:58:46.686120 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:58:46.686087    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0d97cd6_8f3d_4713_80e2_43d8be42fc6c.slice/crio-ecf2ea9ba4b5d4819f026e04f4b561c25e422f9b809a95a095d10553c9847f2b WatchSource:0}: Error finding container ecf2ea9ba4b5d4819f026e04f4b561c25e422f9b809a95a095d10553c9847f2b: Status 404 returned error can't find the container with id ecf2ea9ba4b5d4819f026e04f4b561c25e422f9b809a95a095d10553c9847f2b
Jul 15 20:58:47.457872 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:47.457832    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zb9gv" event={"ID":"b0d97cd6-8f3d-4713-80e2-43d8be42fc6c","Type":"ContainerStarted","Data":"ecf2ea9ba4b5d4819f026e04f4b561c25e422f9b809a95a095d10553c9847f2b"}
Jul 15 20:58:48.463028 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:48.462992    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zb9gv" event={"ID":"b0d97cd6-8f3d-4713-80e2-43d8be42fc6c","Type":"ContainerStarted","Data":"9593b02095d335e88fd0c0809860c9d8946d6dfbc601929e5ebaf29aa5d785fd"}
Jul 15 20:58:48.482956 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:48.482889    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zb9gv" podStartSLOduration=1.357417312 podStartE2EDuration="2.48287341s" totalImagesPullingTime="1.125456098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:58:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:58:48.481880262 +0000 UTC m=+661.630918319" watchObservedRunningTime="2026-07-15 20:58:48.48287341 +0000 UTC m=+661.631911470"
Jul 15 20:58:56.547077 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:56.547029    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zb9gv"
Jul 15 20:58:56.547482 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:56.547125    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zb9gv"
Jul 15 20:58:56.559722 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:56.559698    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zb9gv"
Jul 15 20:58:57.514966 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:58:57.514936    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zb9gv"
Jul 15 20:59:02.345828 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:59:02.345788    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-xfcxw"]
Jul 15 20:59:02.350592 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:59:02.350570    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xfcxw"
Jul 15 20:59:02.354123 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:59:02.354101    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 15 20:59:02.361400 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:59:02.361377    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-xfcxw"]
Jul 15 20:59:02.388069 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:59:02.388042    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:59:02.447982 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:59:02.447954    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b25440fc-4984-4e23-a0c0-dbf490e26002-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xfcxw\" (UID: \"b25440fc-4984-4e23-a0c0-dbf490e26002\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xfcxw"
Jul 15 20:59:02.448155 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:59:02.448002    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b25440fc-4984-4e23-a0c0-dbf490e26002-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xfcxw\" (UID: \"b25440fc-4984-4e23-a0c0-dbf490e26002\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xfcxw"
Jul 15 20:59:02.448155 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:59:02.448026    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b25440fc-4984-4e23-a0c0-dbf490e26002-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xfcxw\" (UID: \"b25440fc-4984-4e23-a0c0-dbf490e26002\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xfcxw"
Jul 15 20:59:02.448326 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:59:02.448166    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b25440fc-4984-4e23-a0c0-dbf490e26002-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xfcxw\" (UID: \"b25440fc-4984-4e23-a0c0-dbf490e26002\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xfcxw"
Jul 15 20:59:02.448326 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:59:02.448204    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b25440fc-4984-4e23-a0c0-dbf490e26002-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xfcxw\" (UID: \"b25440fc-4984-4e23-a0c0-dbf490e26002\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xfcxw"
Jul 15 20:59:02.448326 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:59:02.448233    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg2lj\" (UniqueName: \"kubernetes.io/projected/b25440fc-4984-4e23-a0c0-dbf490e26002-kube-api-access-zg2lj\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xfcxw\" (UID: \"b25440fc-4984-4e23-a0c0-dbf490e26002\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xfcxw"
Jul 15 20:59:02.549711 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:59:02.549674    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b25440fc-4984-4e23-a0c0-dbf490e26002-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xfcxw\" (UID: \"b25440fc-4984-4e23-a0c0-dbf490e26002\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xfcxw"
Jul 15 20:59:02.549711 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:59:02.549708    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b25440fc-4984-4e23-a0c0-dbf490e26002-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xfcxw\" (UID: \"b25440fc-4984-4e23-a0c0-dbf490e26002\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xfcxw"
Jul 15 20:59:02.549974 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:59:02.549726    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg2lj\" (UniqueName: \"kubernetes.io/projected/b25440fc-4984-4e23-a0c0-dbf490e26002-kube-api-access-zg2lj\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xfcxw\" (UID: \"b25440fc-4984-4e23-a0c0-dbf490e26002\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xfcxw"
Jul 15 20:59:02.549974 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:59:02.549746    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b25440fc-4984-4e23-a0c0-dbf490e26002-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xfcxw\" (UID: \"b25440fc-4984-4e23-a0c0-dbf490e26002\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xfcxw"
Jul 15 20:59:02.549974 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:59:02.549788    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b25440fc-4984-4e23-a0c0-dbf490e26002-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xfcxw\" (UID: \"b25440fc-4984-4e23-a0c0-dbf490e26002\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xfcxw"
Jul 15 20:59:02.549974 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:59:02.549816    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b25440fc-4984-4e23-a0c0-dbf490e26002-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xfcxw\" (UID: \"b25440fc-4984-4e23-a0c0-dbf490e26002\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xfcxw"
Jul 15 20:59:02.550182 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:59:02.550163    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b25440fc-4984-4e23-a0c0-dbf490e26002-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xfcxw\" (UID: \"b25440fc-4984-4e23-a0c0-dbf490e26002\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xfcxw"
Jul 15 20:59:02.550240 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:59:02.550211    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b25440fc-4984-4e23-a0c0-dbf490e26002-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xfcxw\" (UID: \"b25440fc-4984-4e23-a0c0-dbf490e26002\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xfcxw"
Jul 15 20:59:02.550317 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:59:02.550296    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b25440fc-4984-4e23-a0c0-dbf490e26002-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xfcxw\" (UID: \"b25440fc-4984-4e23-a0c0-dbf490e26002\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xfcxw"
Jul 15 20:59:02.552172 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:59:02.552151    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b25440fc-4984-4e23-a0c0-dbf490e26002-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xfcxw\" (UID: \"b25440fc-4984-4e23-a0c0-dbf490e26002\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xfcxw"
Jul 15 20:59:02.552532 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:59:02.552506    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b25440fc-4984-4e23-a0c0-dbf490e26002-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xfcxw\" (UID: \"b25440fc-4984-4e23-a0c0-dbf490e26002\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xfcxw"
Jul 15 20:59:02.558589 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:59:02.558568    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg2lj\" (UniqueName: \"kubernetes.io/projected/b25440fc-4984-4e23-a0c0-dbf490e26002-kube-api-access-zg2lj\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xfcxw\" (UID: \"b25440fc-4984-4e23-a0c0-dbf490e26002\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xfcxw"
Jul 15 20:59:02.660883 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:59:02.660782    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xfcxw"
Jul 15 20:59:02.794136 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:59:02.794108    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-xfcxw"]
Jul 15 20:59:02.796705 ip-10-0-137-234 kubenswrapper[2566]: W0715 20:59:02.796675    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb25440fc_4984_4e23_a0c0_dbf490e26002.slice/crio-25ec2a71b17f1c6e416a6abf816779928cdf28a8c267ad43477f7778908038c3 WatchSource:0}: Error finding container 25ec2a71b17f1c6e416a6abf816779928cdf28a8c267ad43477f7778908038c3: Status 404 returned error can't find the container with id 25ec2a71b17f1c6e416a6abf816779928cdf28a8c267ad43477f7778908038c3
Jul 15 20:59:03.530288 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:59:03.530255    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xfcxw" event={"ID":"b25440fc-4984-4e23-a0c0-dbf490e26002","Type":"ContainerStarted","Data":"c9efbdead88d03afbfb0baf98e0d9104b10d623d1240fdc1c908024a41c5c302"}
Jul 15 20:59:03.530711 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:59:03.530303    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xfcxw" event={"ID":"b25440fc-4984-4e23-a0c0-dbf490e26002","Type":"ContainerStarted","Data":"25ec2a71b17f1c6e416a6abf816779928cdf28a8c267ad43477f7778908038c3"}
Jul 15 20:59:03.578125 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:59:03.578077    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xfcxw" podStartSLOduration=1.389898993 podStartE2EDuration="1.578060486s" totalImagesPullingTime="188.161493ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:59:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:59:03.576000052 +0000 UTC m=+676.725038108" watchObservedRunningTime="2026-07-15 20:59:03.578060486 +0000 UTC m=+676.727098542"
Jul 15 20:59:12.661429 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:59:12.661393    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xfcxw"
Jul 15 20:59:12.661837 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:59:12.661461    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xfcxw"
Jul 15 20:59:12.673858 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:59:12.673835    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xfcxw"
Jul 15 20:59:13.582795 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:59:13.582766    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xfcxw"
Jul 15 20:59:23.880599 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:59:23.880563    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:59:33.277684 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:59:33.277649    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:59:51.641782 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:59:51.641688    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 20:59:59.387051 ip-10-0-137-234 kubenswrapper[2566]: I0715 20:59:59.387011    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:00:00.144291 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:00:00.144255    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29735820-gm549"]
Jul 15 21:00:00.146708 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:00:00.146691    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735820-gm549"
Jul 15 21:00:00.159514 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:00:00.159493    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zsrf5\""
Jul 15 21:00:00.165815 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:00:00.165800    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735820-gm549"
Jul 15 21:00:00.174192 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:00:00.174169    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29735820-gm549"]
Jul 15 21:00:00.314504 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:00:00.314470    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29735820-gm549"]
Jul 15 21:00:00.320002 ip-10-0-137-234 kubenswrapper[2566]: W0715 21:00:00.319964    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62fd87a2_6d1e_4bac_8354_862921c2587a.slice/crio-29ee9d5bd62b5fc58937afb442853b87926d311417424b275a7798ffc13cacbc WatchSource:0}: Error finding container 29ee9d5bd62b5fc58937afb442853b87926d311417424b275a7798ffc13cacbc: Status 404 returned error can't find the container with id 29ee9d5bd62b5fc58937afb442853b87926d311417424b275a7798ffc13cacbc
Jul 15 21:00:00.322207 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:00:00.322191    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 15 21:00:00.773636 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:00:00.773593    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735820-gm549" event={"ID":"62fd87a2-6d1e-4bac-8354-862921c2587a","Type":"ContainerStarted","Data":"29ee9d5bd62b5fc58937afb442853b87926d311417424b275a7798ffc13cacbc"}
Jul 15 21:00:05.801448 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:00:05.801420    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735820-gm549" event={"ID":"62fd87a2-6d1e-4bac-8354-862921c2587a","Type":"ContainerStarted","Data":"2cac91bf82804f2cb1cdd7a2d0edba96ec7ce921655106886550d6fbec07f600"}
Jul 15 21:00:06.806861 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:00:06.806827    2566 generic.go:356] "Generic (PLEG): container finished" podID="62fd87a2-6d1e-4bac-8354-862921c2587a" containerID="2cac91bf82804f2cb1cdd7a2d0edba96ec7ce921655106886550d6fbec07f600" exitCode=0
Jul 15 21:00:06.807291 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:00:06.806935    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735820-gm549" event={"ID":"62fd87a2-6d1e-4bac-8354-862921c2587a","Type":"ContainerDied","Data":"2cac91bf82804f2cb1cdd7a2d0edba96ec7ce921655106886550d6fbec07f600"}
Jul 15 21:00:06.940289 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:00:06.940257    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735820-gm549"
Jul 15 21:00:07.812287 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:00:07.812259    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735820-gm549"
Jul 15 21:00:07.812685 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:00:07.812309    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735820-gm549" event={"ID":"62fd87a2-6d1e-4bac-8354-862921c2587a","Type":"ContainerDied","Data":"29ee9d5bd62b5fc58937afb442853b87926d311417424b275a7798ffc13cacbc"}
Jul 15 21:00:07.812685 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:00:07.812366    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29ee9d5bd62b5fc58937afb442853b87926d311417424b275a7798ffc13cacbc"
Jul 15 21:01:08.208450 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:01:08.208365    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:01:08.305559 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:01:08.305524    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:01:09.509112 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:01:09.509078    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:01:10.222991 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:01:10.222955    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:02:30.851401 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:02:30.851354    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:02:30.963362 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:02:30.963313    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-567446f9b-8wqwj"]
Jul 15 21:02:30.963555 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:02:30.963522    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-567446f9b-8wqwj" podUID="e9593db8-063d-4d3e-8420-8df1a687f3d5" containerName="manager" containerID="cri-o://86efb21a65b88324da0ef11f1eeb2e1c6b1919d5d40e1ea24e55064a34a18019" gracePeriod=30
Jul 15 21:02:31.209313 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:02:31.209289    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-567446f9b-8wqwj"
Jul 15 21:02:31.350179 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:02:31.350151    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e9593db8-063d-4d3e-8420-8df1a687f3d5-webhook-cert\") pod \"e9593db8-063d-4d3e-8420-8df1a687f3d5\" (UID: \"e9593db8-063d-4d3e-8420-8df1a687f3d5\") "
Jul 15 21:02:31.350358 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:02:31.350214    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nc4lz\" (UniqueName: \"kubernetes.io/projected/e9593db8-063d-4d3e-8420-8df1a687f3d5-kube-api-access-nc4lz\") pod \"e9593db8-063d-4d3e-8420-8df1a687f3d5\" (UID: \"e9593db8-063d-4d3e-8420-8df1a687f3d5\") "
Jul 15 21:02:31.352333 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:02:31.352297    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9593db8-063d-4d3e-8420-8df1a687f3d5-kube-api-access-nc4lz" (OuterVolumeSpecName: "kube-api-access-nc4lz") pod "e9593db8-063d-4d3e-8420-8df1a687f3d5" (UID: "e9593db8-063d-4d3e-8420-8df1a687f3d5"). InnerVolumeSpecName "kube-api-access-nc4lz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 21:02:31.352440 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:02:31.352346    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9593db8-063d-4d3e-8420-8df1a687f3d5-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "e9593db8-063d-4d3e-8420-8df1a687f3d5" (UID: "e9593db8-063d-4d3e-8420-8df1a687f3d5"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 21:02:31.421580 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:02:31.421480    2566 generic.go:356] "Generic (PLEG): container finished" podID="e9593db8-063d-4d3e-8420-8df1a687f3d5" containerID="86efb21a65b88324da0ef11f1eeb2e1c6b1919d5d40e1ea24e55064a34a18019" exitCode=0
Jul 15 21:02:31.421580 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:02:31.421534    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-567446f9b-8wqwj"
Jul 15 21:02:31.421805 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:02:31.421539    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-567446f9b-8wqwj" event={"ID":"e9593db8-063d-4d3e-8420-8df1a687f3d5","Type":"ContainerDied","Data":"86efb21a65b88324da0ef11f1eeb2e1c6b1919d5d40e1ea24e55064a34a18019"}
Jul 15 21:02:31.421805 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:02:31.421663    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-567446f9b-8wqwj" event={"ID":"e9593db8-063d-4d3e-8420-8df1a687f3d5","Type":"ContainerDied","Data":"99ebe426e3103f2f4067039a5e70f6b4f1db8cc3255de27a3437a308386b1ed0"}
Jul 15 21:02:31.421805 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:02:31.421692    2566 scope.go:117] "RemoveContainer" containerID="86efb21a65b88324da0ef11f1eeb2e1c6b1919d5d40e1ea24e55064a34a18019"
Jul 15 21:02:31.430367 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:02:31.430347    2566 scope.go:117] "RemoveContainer" containerID="86efb21a65b88324da0ef11f1eeb2e1c6b1919d5d40e1ea24e55064a34a18019"
Jul 15 21:02:31.430624 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:02:31.430606    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86efb21a65b88324da0ef11f1eeb2e1c6b1919d5d40e1ea24e55064a34a18019\": container with ID starting with 86efb21a65b88324da0ef11f1eeb2e1c6b1919d5d40e1ea24e55064a34a18019 not found: ID does not exist" containerID="86efb21a65b88324da0ef11f1eeb2e1c6b1919d5d40e1ea24e55064a34a18019"
Jul 15 21:02:31.430678 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:02:31.430632    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86efb21a65b88324da0ef11f1eeb2e1c6b1919d5d40e1ea24e55064a34a18019"} err="failed to get container status \"86efb21a65b88324da0ef11f1eeb2e1c6b1919d5d40e1ea24e55064a34a18019\": rpc error: code = NotFound desc = could not find container \"86efb21a65b88324da0ef11f1eeb2e1c6b1919d5d40e1ea24e55064a34a18019\": container with ID starting with 86efb21a65b88324da0ef11f1eeb2e1c6b1919d5d40e1ea24e55064a34a18019 not found: ID does not exist"
Jul 15 21:02:31.444100 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:02:31.444077    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-567446f9b-8wqwj"]
Jul 15 21:02:31.448381 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:02:31.448356    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-567446f9b-8wqwj"]
Jul 15 21:02:31.451234 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:02:31.451213    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc4lz\" (UniqueName: \"kubernetes.io/projected/e9593db8-063d-4d3e-8420-8df1a687f3d5-kube-api-access-nc4lz\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:02:31.451318 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:02:31.451236    2566 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e9593db8-063d-4d3e-8420-8df1a687f3d5-webhook-cert\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:02:31.756668 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:02:31.756640    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9593db8-063d-4d3e-8420-8df1a687f3d5" path="/var/lib/kubelet/pods/e9593db8-063d-4d3e-8420-8df1a687f3d5/volumes"
Jul 15 21:02:32.154558 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:02:32.154467    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:02:32.285568 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:02:32.285534    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-567446f9b-dgm4c"]
Jul 15 21:02:32.285855 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:02:32.285844    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9593db8-063d-4d3e-8420-8df1a687f3d5" containerName="manager"
Jul 15 21:02:32.285922 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:02:32.285856    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9593db8-063d-4d3e-8420-8df1a687f3d5" containerName="manager"
Jul 15 21:02:32.285922 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:02:32.285882    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62fd87a2-6d1e-4bac-8354-862921c2587a" containerName="cleanup"
Jul 15 21:02:32.285922 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:02:32.285888    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="62fd87a2-6d1e-4bac-8354-862921c2587a" containerName="cleanup"
Jul 15 21:02:32.286016 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:02:32.285961    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9593db8-063d-4d3e-8420-8df1a687f3d5" containerName="manager"
Jul 15 21:02:32.286016 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:02:32.285970    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="62fd87a2-6d1e-4bac-8354-862921c2587a" containerName="cleanup"
Jul 15 21:02:32.288482 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:02:32.288465    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-567446f9b-dgm4c"
Jul 15 21:02:32.293591 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:02:32.293545    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 15 21:02:32.293591 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:02:32.293553    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 15 21:02:32.293591 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:02:32.293553    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-s478v\""
Jul 15 21:02:32.300481 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:02:32.300449    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-567446f9b-dgm4c"]
Jul 15 21:02:32.360239 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:02:32.360213    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v467k\" (UniqueName: \"kubernetes.io/projected/73b59907-4d42-4445-b7b9-91a80790a8ba-kube-api-access-v467k\") pod \"maas-controller-567446f9b-dgm4c\" (UID: \"73b59907-4d42-4445-b7b9-91a80790a8ba\") " pod="opendatahub/maas-controller-567446f9b-dgm4c"
Jul 15 21:02:32.360367 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:02:32.360259    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/73b59907-4d42-4445-b7b9-91a80790a8ba-webhook-cert\") pod \"maas-controller-567446f9b-dgm4c\" (UID: \"73b59907-4d42-4445-b7b9-91a80790a8ba\") " pod="opendatahub/maas-controller-567446f9b-dgm4c"
Jul 15 21:02:32.460721 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:02:32.460691    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v467k\" (UniqueName: \"kubernetes.io/projected/73b59907-4d42-4445-b7b9-91a80790a8ba-kube-api-access-v467k\") pod \"maas-controller-567446f9b-dgm4c\" (UID: \"73b59907-4d42-4445-b7b9-91a80790a8ba\") " pod="opendatahub/maas-controller-567446f9b-dgm4c"
Jul 15 21:02:32.460886 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:02:32.460746    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/73b59907-4d42-4445-b7b9-91a80790a8ba-webhook-cert\") pod \"maas-controller-567446f9b-dgm4c\" (UID: \"73b59907-4d42-4445-b7b9-91a80790a8ba\") " pod="opendatahub/maas-controller-567446f9b-dgm4c"
Jul 15 21:02:32.463151 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:02:32.463127    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/73b59907-4d42-4445-b7b9-91a80790a8ba-webhook-cert\") pod \"maas-controller-567446f9b-dgm4c\" (UID: \"73b59907-4d42-4445-b7b9-91a80790a8ba\") " pod="opendatahub/maas-controller-567446f9b-dgm4c"
Jul 15 21:02:32.469752 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:02:32.469725    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v467k\" (UniqueName: \"kubernetes.io/projected/73b59907-4d42-4445-b7b9-91a80790a8ba-kube-api-access-v467k\") pod \"maas-controller-567446f9b-dgm4c\" (UID: \"73b59907-4d42-4445-b7b9-91a80790a8ba\") " pod="opendatahub/maas-controller-567446f9b-dgm4c"
Jul 15 21:02:32.630811 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:02:32.630763    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-567446f9b-dgm4c"
Jul 15 21:02:32.764972 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:02:32.764940    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-567446f9b-dgm4c"]
Jul 15 21:02:32.766998 ip-10-0-137-234 kubenswrapper[2566]: W0715 21:02:32.766967    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73b59907_4d42_4445_b7b9_91a80790a8ba.slice/crio-02046d554447a3c9250dccd9415c1af2501db40ce280cf51048b4e98086d48ef WatchSource:0}: Error finding container 02046d554447a3c9250dccd9415c1af2501db40ce280cf51048b4e98086d48ef: Status 404 returned error can't find the container with id 02046d554447a3c9250dccd9415c1af2501db40ce280cf51048b4e98086d48ef
Jul 15 21:02:33.435640 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:02:33.435601    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-567446f9b-dgm4c" event={"ID":"73b59907-4d42-4445-b7b9-91a80790a8ba","Type":"ContainerStarted","Data":"f9171c6dfb521ebd88c624baa84c3d0564abea32aa69e8b853d3983d5e53b241"}
Jul 15 21:02:33.436046 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:02:33.435664    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-567446f9b-dgm4c" event={"ID":"73b59907-4d42-4445-b7b9-91a80790a8ba","Type":"ContainerStarted","Data":"02046d554447a3c9250dccd9415c1af2501db40ce280cf51048b4e98086d48ef"}
Jul 15 21:02:33.436046 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:02:33.435747    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-567446f9b-dgm4c"
Jul 15 21:02:33.456459 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:02:33.456415    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-567446f9b-dgm4c" podStartSLOduration=0.967136303 podStartE2EDuration="1.456400817s" totalImagesPullingTime="489.264514ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:02:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:02:33.455140596 +0000 UTC m=+886.604178694" watchObservedRunningTime="2026-07-15 21:02:33.456400817 +0000 UTC m=+886.605438874"
Jul 15 21:02:34.002373 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:02:34.002328    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:02:44.442509 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:02:44.442474    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-567446f9b-dgm4c"
Jul 15 21:02:47.679428 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:02:47.679398    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-652lk_f9a159bb-39c4-4cca-92ff-1c74d739e311/ovn-acl-logging/0.log"
Jul 15 21:02:47.681344 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:02:47.681322    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-652lk_f9a159bb-39c4-4cca-92ff-1c74d739e311/ovn-acl-logging/0.log"
Jul 15 21:02:55.616472 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:02:55.616434    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:02:55.677218 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:02:55.677181    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:03:18.289599 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:03:18.289567    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:03:18.387774 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:03:18.387736    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:03:18.533147 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:03:18.533111    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:03:24.587088 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:03:24.587053    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:03:26.617775 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:03:26.617743    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:03:31.136541 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:03:31.136499    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:03:31.241100 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:03:31.241048    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:03:42.867374 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:03:42.867337    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:06:26.004690 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:06:26.004645    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:06:51.759738 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:06:51.759697    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:07:31.175927 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:07:31.175823    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5spsp"]
Jul 15 21:07:31.176514 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:07:31.176092    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5spsp" podUID="d1e08988-fd2c-4558-855e-cbdeb7d6f3f4" containerName="manager" containerID="cri-o://1e23b67e08637289fcd329fabbfeec503152ac14fab2706251997a426053775c" gracePeriod=10
Jul 15 21:07:34.621213 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:07:34.621190    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5spsp"
Jul 15 21:07:34.629875 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:07:34.629853    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d1e08988-fd2c-4558-855e-cbdeb7d6f3f4-extensions-socket-volume\") pod \"d1e08988-fd2c-4558-855e-cbdeb7d6f3f4\" (UID: \"d1e08988-fd2c-4558-855e-cbdeb7d6f3f4\") "
Jul 15 21:07:34.630024 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:07:34.629982    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-skbqc\" (UniqueName: \"kubernetes.io/projected/d1e08988-fd2c-4558-855e-cbdeb7d6f3f4-kube-api-access-skbqc\") pod \"d1e08988-fd2c-4558-855e-cbdeb7d6f3f4\" (UID: \"d1e08988-fd2c-4558-855e-cbdeb7d6f3f4\") "
Jul 15 21:07:34.630197 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:07:34.630173    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1e08988-fd2c-4558-855e-cbdeb7d6f3f4-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d1e08988-fd2c-4558-855e-cbdeb7d6f3f4" (UID: "d1e08988-fd2c-4558-855e-cbdeb7d6f3f4"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:07:34.631860 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:07:34.631836    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1e08988-fd2c-4558-855e-cbdeb7d6f3f4-kube-api-access-skbqc" (OuterVolumeSpecName: "kube-api-access-skbqc") pod "d1e08988-fd2c-4558-855e-cbdeb7d6f3f4" (UID: "d1e08988-fd2c-4558-855e-cbdeb7d6f3f4"). InnerVolumeSpecName "kube-api-access-skbqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 21:07:34.730844 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:07:34.730758    2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d1e08988-fd2c-4558-855e-cbdeb7d6f3f4-extensions-socket-volume\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:07:34.730844 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:07:34.730788    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skbqc\" (UniqueName: \"kubernetes.io/projected/d1e08988-fd2c-4558-855e-cbdeb7d6f3f4-kube-api-access-skbqc\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:07:34.769871 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:07:34.769834    2566 generic.go:356] "Generic (PLEG): container finished" podID="d1e08988-fd2c-4558-855e-cbdeb7d6f3f4" containerID="1e23b67e08637289fcd329fabbfeec503152ac14fab2706251997a426053775c" exitCode=0
Jul 15 21:07:34.770028 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:07:34.769882    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5spsp"
Jul 15 21:07:34.770028 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:07:34.769916    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5spsp" event={"ID":"d1e08988-fd2c-4558-855e-cbdeb7d6f3f4","Type":"ContainerDied","Data":"1e23b67e08637289fcd329fabbfeec503152ac14fab2706251997a426053775c"}
Jul 15 21:07:34.770028 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:07:34.769971    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5spsp" event={"ID":"d1e08988-fd2c-4558-855e-cbdeb7d6f3f4","Type":"ContainerDied","Data":"819f07944f6561ef527ebbc39447e0c7f04c9486e9c7d1d119906cbbae95c03c"}
Jul 15 21:07:34.770028 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:07:34.769992    2566 scope.go:117] "RemoveContainer" containerID="1e23b67e08637289fcd329fabbfeec503152ac14fab2706251997a426053775c"
Jul 15 21:07:34.779262 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:07:34.779247    2566 scope.go:117] "RemoveContainer" containerID="1e23b67e08637289fcd329fabbfeec503152ac14fab2706251997a426053775c"
Jul 15 21:07:34.779530 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:07:34.779514    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e23b67e08637289fcd329fabbfeec503152ac14fab2706251997a426053775c\": container with ID starting with 1e23b67e08637289fcd329fabbfeec503152ac14fab2706251997a426053775c not found: ID does not exist" containerID="1e23b67e08637289fcd329fabbfeec503152ac14fab2706251997a426053775c"
Jul 15 21:07:34.779576 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:07:34.779537    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e23b67e08637289fcd329fabbfeec503152ac14fab2706251997a426053775c"} err="failed to get container status \"1e23b67e08637289fcd329fabbfeec503152ac14fab2706251997a426053775c\": rpc error: code = NotFound desc = could not find container \"1e23b67e08637289fcd329fabbfeec503152ac14fab2706251997a426053775c\": container with ID starting with 1e23b67e08637289fcd329fabbfeec503152ac14fab2706251997a426053775c not found: ID does not exist"
Jul 15 21:07:34.793717 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:07:34.793683    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5spsp"]
Jul 15 21:07:34.797940 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:07:34.797920    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5spsp"]
Jul 15 21:07:35.756790 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:07:35.756756    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1e08988-fd2c-4558-855e-cbdeb7d6f3f4" path="/var/lib/kubelet/pods/d1e08988-fd2c-4558-855e-cbdeb7d6f3f4/volumes"
Jul 15 21:07:47.705054 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:07:47.705028    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-652lk_f9a159bb-39c4-4cca-92ff-1c74d739e311/ovn-acl-logging/0.log"
Jul 15 21:07:47.707285 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:07:47.707262    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-652lk_f9a159bb-39c4-4cca-92ff-1c74d739e311/ovn-acl-logging/0.log"
Jul 15 21:08:32.422215 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:08:32.422177    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l7nv5"]
Jul 15 21:08:32.423704 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:08:32.422516    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1e08988-fd2c-4558-855e-cbdeb7d6f3f4" containerName="manager"
Jul 15 21:08:32.423704 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:08:32.422527    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1e08988-fd2c-4558-855e-cbdeb7d6f3f4" containerName="manager"
Jul 15 21:08:32.423704 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:08:32.422581    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1e08988-fd2c-4558-855e-cbdeb7d6f3f4" containerName="manager"
Jul 15 21:08:32.424426 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:08:32.424409    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l7nv5"
Jul 15 21:08:32.427405 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:08:32.427385    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-k9x7l\""
Jul 15 21:08:32.436468 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:08:32.436444    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l7nv5"]
Jul 15 21:08:32.611958 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:08:32.611916    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2kxw\" (UniqueName: \"kubernetes.io/projected/856d10c0-5cc3-4e42-a3c1-9274c31e5e01-kube-api-access-f2kxw\") pod \"kuadrant-operator-controller-manager-86878c769f-l7nv5\" (UID: \"856d10c0-5cc3-4e42-a3c1-9274c31e5e01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l7nv5"
Jul 15 21:08:32.612144 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:08:32.611964    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/856d10c0-5cc3-4e42-a3c1-9274c31e5e01-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-l7nv5\" (UID: \"856d10c0-5cc3-4e42-a3c1-9274c31e5e01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l7nv5"
Jul 15 21:08:32.713101 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:08:32.713061    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2kxw\" (UniqueName: \"kubernetes.io/projected/856d10c0-5cc3-4e42-a3c1-9274c31e5e01-kube-api-access-f2kxw\") pod \"kuadrant-operator-controller-manager-86878c769f-l7nv5\" (UID: \"856d10c0-5cc3-4e42-a3c1-9274c31e5e01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l7nv5"
Jul 15 21:08:32.713291 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:08:32.713107    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/856d10c0-5cc3-4e42-a3c1-9274c31e5e01-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-l7nv5\" (UID: \"856d10c0-5cc3-4e42-a3c1-9274c31e5e01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l7nv5"
Jul 15 21:08:32.713521 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:08:32.713497    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/856d10c0-5cc3-4e42-a3c1-9274c31e5e01-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-l7nv5\" (UID: \"856d10c0-5cc3-4e42-a3c1-9274c31e5e01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l7nv5"
Jul 15 21:08:32.726417 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:08:32.726392    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2kxw\" (UniqueName: \"kubernetes.io/projected/856d10c0-5cc3-4e42-a3c1-9274c31e5e01-kube-api-access-f2kxw\") pod \"kuadrant-operator-controller-manager-86878c769f-l7nv5\" (UID: \"856d10c0-5cc3-4e42-a3c1-9274c31e5e01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l7nv5"
Jul 15 21:08:32.734195 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:08:32.734174    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l7nv5"
Jul 15 21:08:32.884860 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:08:32.884834    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l7nv5"]
Jul 15 21:08:32.886992 ip-10-0-137-234 kubenswrapper[2566]: W0715 21:08:32.886955    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod856d10c0_5cc3_4e42_a3c1_9274c31e5e01.slice/crio-de16d801fdbc8ca2a28662f3008b11787a61a5c2ea6ddb87f2a142fe1f72250c WatchSource:0}: Error finding container de16d801fdbc8ca2a28662f3008b11787a61a5c2ea6ddb87f2a142fe1f72250c: Status 404 returned error can't find the container with id de16d801fdbc8ca2a28662f3008b11787a61a5c2ea6ddb87f2a142fe1f72250c
Jul 15 21:08:33.012445 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:08:33.012346    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l7nv5" event={"ID":"856d10c0-5cc3-4e42-a3c1-9274c31e5e01","Type":"ContainerStarted","Data":"aed25a04111aa918df6219b599c221d087061a135603ef8efbc87808ed06d53e"}
Jul 15 21:08:33.012445 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:08:33.012402    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l7nv5" event={"ID":"856d10c0-5cc3-4e42-a3c1-9274c31e5e01","Type":"ContainerStarted","Data":"de16d801fdbc8ca2a28662f3008b11787a61a5c2ea6ddb87f2a142fe1f72250c"}
Jul 15 21:08:33.012650 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:08:33.012499    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l7nv5"
Jul 15 21:08:33.037456 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:08:33.037409    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l7nv5" podStartSLOduration=1.037393722 podStartE2EDuration="1.037393722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:08:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:08:33.0356801 +0000 UTC m=+1246.184718158" watchObservedRunningTime="2026-07-15 21:08:33.037393722 +0000 UTC m=+1246.186431827"
Jul 15 21:08:44.019721 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:08:44.019639    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l7nv5"
Jul 15 21:09:18.313070 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:09:18.313038    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:12:13.900175 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:12:13.900143    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:12:47.730696 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:12:47.730665    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-652lk_f9a159bb-39c4-4cca-92ff-1c74d739e311/ovn-acl-logging/0.log"
Jul 15 21:12:47.733623 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:12:47.733592    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-652lk_f9a159bb-39c4-4cca-92ff-1c74d739e311/ovn-acl-logging/0.log"
Jul 15 21:14:32.619506 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:14:32.619468    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:14:40.683419 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:14:40.683323    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:14:48.953913 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:14:48.953863    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:14:51.326461 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:14:51.326424    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:14:54.119545 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:14:54.119505    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:14:57.233020 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:14:57.232987    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:14:57.329012 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:14:57.328973    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:14:57.724730 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:14:57.724697    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:15:00.147013 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:00.146980    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29735835-x4vms"]
Jul 15 21:15:00.149856 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:00.149839    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735835-x4vms"
Jul 15 21:15:00.153047 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:00.153027    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zsrf5\""
Jul 15 21:15:00.159401 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:00.159380    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735835-x4vms"
Jul 15 21:15:00.172132 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:00.172102    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29735835-x4vms"]
Jul 15 21:15:00.293377 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:00.293345    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29735835-x4vms"]
Jul 15 21:15:00.296543 ip-10-0-137-234 kubenswrapper[2566]: W0715 21:15:00.296518    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf021cabb_31c6_49c5_8fee_9574df40e4a0.slice/crio-3029c70d66ad5a3ada77589e52cb4181abdf3550f003b4f3ae608d47f868bc8b WatchSource:0}: Error finding container 3029c70d66ad5a3ada77589e52cb4181abdf3550f003b4f3ae608d47f868bc8b: Status 404 returned error can't find the container with id 3029c70d66ad5a3ada77589e52cb4181abdf3550f003b4f3ae608d47f868bc8b
Jul 15 21:15:00.744583 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:00.744543    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735835-x4vms" event={"ID":"f021cabb-31c6-49c5-8fee-9574df40e4a0","Type":"ContainerStarted","Data":"513b81d1577b0282bacaeb3197be02a95fd67cd21e61d76ec4cc132344289e05"}
Jul 15 21:15:00.744825 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:00.744636    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735835-x4vms" event={"ID":"f021cabb-31c6-49c5-8fee-9574df40e4a0","Type":"ContainerStarted","Data":"3029c70d66ad5a3ada77589e52cb4181abdf3550f003b4f3ae608d47f868bc8b"}
Jul 15 21:15:00.762584 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:00.762536    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735835-x4vms" podStartSLOduration=0.762522235 podStartE2EDuration="762.522235ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:15:00.760987414 +0000 UTC m=+1633.910025465" watchObservedRunningTime="2026-07-15 21:15:00.762522235 +0000 UTC m=+1633.911560292"
Jul 15 21:15:01.752619 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:01.752583    2566 generic.go:356] "Generic (PLEG): container finished" podID="f021cabb-31c6-49c5-8fee-9574df40e4a0" containerID="513b81d1577b0282bacaeb3197be02a95fd67cd21e61d76ec4cc132344289e05" exitCode=0
Jul 15 21:15:01.756578 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:01.756548    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735835-x4vms" event={"ID":"f021cabb-31c6-49c5-8fee-9574df40e4a0","Type":"ContainerDied","Data":"513b81d1577b0282bacaeb3197be02a95fd67cd21e61d76ec4cc132344289e05"}
Jul 15 21:15:02.888112 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:02.888086    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735835-x4vms"
Jul 15 21:15:03.762604 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:03.762563    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735835-x4vms" event={"ID":"f021cabb-31c6-49c5-8fee-9574df40e4a0","Type":"ContainerDied","Data":"3029c70d66ad5a3ada77589e52cb4181abdf3550f003b4f3ae608d47f868bc8b"}
Jul 15 21:15:03.762772 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:03.762615    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3029c70d66ad5a3ada77589e52cb4181abdf3550f003b4f3ae608d47f868bc8b"
Jul 15 21:15:03.762772 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:03.762582    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735835-x4vms"
Jul 15 21:15:03.837687 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:03.837654    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:15:03.940215 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:03.940184    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:15:04.217911 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:04.217866    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt"]
Jul 15 21:15:04.218238 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:04.218225    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f021cabb-31c6-49c5-8fee-9574df40e4a0" containerName="cleanup"
Jul 15 21:15:04.218289 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:04.218240    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f021cabb-31c6-49c5-8fee-9574df40e4a0" containerName="cleanup"
Jul 15 21:15:04.218324 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:04.218297    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f021cabb-31c6-49c5-8fee-9574df40e4a0" containerName="cleanup"
Jul 15 21:15:04.221182 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:04.221166    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt"
Jul 15 21:15:04.225325 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:04.225306    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-8143c542-openshift-default-dockercfg-t4rlq\""
Jul 15 21:15:04.225683 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:04.225670    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 15 21:15:04.243731 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:04.243708    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:15:04.245045 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:04.245024    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt"]
Jul 15 21:15:04.269884 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:04.269855    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-credential-socket\") pod \"e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt\" (UID: \"0f7d9980-d5ed-4368-9f06-4eb0885b2e9d\") " pod="openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt"
Jul 15 21:15:04.270073 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:04.269887    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzhbw\" (UniqueName: \"kubernetes.io/projected/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-kube-api-access-nzhbw\") pod \"e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt\" (UID: \"0f7d9980-d5ed-4368-9f06-4eb0885b2e9d\") " pod="openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt"
Jul 15 21:15:04.270073 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:04.269984    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-workload-certs\") pod \"e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt\" (UID: \"0f7d9980-d5ed-4368-9f06-4eb0885b2e9d\") " pod="openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt"
Jul 15 21:15:04.270073 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:04.270039    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-istiod-ca-cert\") pod \"e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt\" (UID: \"0f7d9980-d5ed-4368-9f06-4eb0885b2e9d\") " pod="openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt"
Jul 15 21:15:04.270209 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:04.270104    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-istio-envoy\") pod \"e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt\" (UID: \"0f7d9980-d5ed-4368-9f06-4eb0885b2e9d\") " pod="openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt"
Jul 15 21:15:04.270209 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:04.270132    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-istio-data\") pod \"e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt\" (UID: \"0f7d9980-d5ed-4368-9f06-4eb0885b2e9d\") " pod="openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt"
Jul 15 21:15:04.270209 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:04.270179    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-istio-podinfo\") pod \"e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt\" (UID: \"0f7d9980-d5ed-4368-9f06-4eb0885b2e9d\") " pod="openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt"
Jul 15 21:15:04.270340 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:04.270214    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-istio-token\") pod \"e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt\" (UID: \"0f7d9980-d5ed-4368-9f06-4eb0885b2e9d\") " pod="openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt"
Jul 15 21:15:04.270340 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:04.270251    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-workload-socket\") pod \"e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt\" (UID: \"0f7d9980-d5ed-4368-9f06-4eb0885b2e9d\") " pod="openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt"
Jul 15 21:15:04.360019 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:04.359972    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:15:04.371111 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:04.371086    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-workload-socket\") pod \"e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt\" (UID: \"0f7d9980-d5ed-4368-9f06-4eb0885b2e9d\") " pod="openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt"
Jul 15 21:15:04.371222 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:04.371123    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-credential-socket\") pod \"e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt\" (UID: \"0f7d9980-d5ed-4368-9f06-4eb0885b2e9d\") " pod="openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt"
Jul 15 21:15:04.371222 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:04.371145    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzhbw\" (UniqueName: \"kubernetes.io/projected/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-kube-api-access-nzhbw\") pod \"e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt\" (UID: \"0f7d9980-d5ed-4368-9f06-4eb0885b2e9d\") " pod="openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt"
Jul 15 21:15:04.371222 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:04.371183    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-workload-certs\") pod \"e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt\" (UID: \"0f7d9980-d5ed-4368-9f06-4eb0885b2e9d\") " pod="openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt"
Jul 15 21:15:04.371222 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:04.371216    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-istiod-ca-cert\") pod \"e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt\" (UID: \"0f7d9980-d5ed-4368-9f06-4eb0885b2e9d\") " pod="openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt"
Jul 15 21:15:04.371426 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:04.371271    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-istio-envoy\") pod \"e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt\" (UID: \"0f7d9980-d5ed-4368-9f06-4eb0885b2e9d\") " pod="openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt"
Jul 15 21:15:04.371426 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:04.371295    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-istio-data\") pod \"e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt\" (UID: \"0f7d9980-d5ed-4368-9f06-4eb0885b2e9d\") " pod="openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt"
Jul 15 21:15:04.371426 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:04.371336    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-istio-podinfo\") pod \"e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt\" (UID: \"0f7d9980-d5ed-4368-9f06-4eb0885b2e9d\") " pod="openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt"
Jul 15 21:15:04.371426 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:04.371367    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-istio-token\") pod \"e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt\" (UID: \"0f7d9980-d5ed-4368-9f06-4eb0885b2e9d\") " pod="openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt"
Jul 15 21:15:04.371657 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:04.371563    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-workload-socket\") pod \"e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt\" (UID: \"0f7d9980-d5ed-4368-9f06-4eb0885b2e9d\") " pod="openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt"
Jul 15 21:15:04.371657 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:04.371607    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-workload-certs\") pod \"e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt\" (UID: \"0f7d9980-d5ed-4368-9f06-4eb0885b2e9d\") " pod="openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt"
Jul 15 21:15:04.371762 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:04.371651    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-credential-socket\") pod \"e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt\" (UID: \"0f7d9980-d5ed-4368-9f06-4eb0885b2e9d\") " pod="openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt"
Jul 15 21:15:04.371841 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:04.371809    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-istio-data\") pod \"e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt\" (UID: \"0f7d9980-d5ed-4368-9f06-4eb0885b2e9d\") " pod="openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt"
Jul 15 21:15:04.372030 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:04.372012    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-istiod-ca-cert\") pod \"e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt\" (UID: \"0f7d9980-d5ed-4368-9f06-4eb0885b2e9d\") " pod="openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt"
Jul 15 21:15:04.373556 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:04.373533    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-istio-envoy\") pod \"e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt\" (UID: \"0f7d9980-d5ed-4368-9f06-4eb0885b2e9d\") " pod="openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt"
Jul 15 21:15:04.373788 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:04.373771    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-istio-podinfo\") pod \"e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt\" (UID: \"0f7d9980-d5ed-4368-9f06-4eb0885b2e9d\") " pod="openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt"
Jul 15 21:15:04.399680 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:04.399612    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzhbw\" (UniqueName: \"kubernetes.io/projected/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-kube-api-access-nzhbw\") pod \"e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt\" (UID: \"0f7d9980-d5ed-4368-9f06-4eb0885b2e9d\") " pod="openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt"
Jul 15 21:15:04.400130 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:04.400104    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-istio-token\") pod \"e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt\" (UID: \"0f7d9980-d5ed-4368-9f06-4eb0885b2e9d\") " pod="openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt"
Jul 15 21:15:04.530961 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:04.530924    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt"
Jul 15 21:15:04.668439 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:04.668365    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt"]
Jul 15 21:15:04.670481 ip-10-0-137-234 kubenswrapper[2566]: W0715 21:15:04.670453    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f7d9980_d5ed_4368_9f06_4eb0885b2e9d.slice/crio-f47b50e114df46286b4bae10faa5738d74c6eafe33d04608e1a559292b0a6f08 WatchSource:0}: Error finding container f47b50e114df46286b4bae10faa5738d74c6eafe33d04608e1a559292b0a6f08: Status 404 returned error can't find the container with id f47b50e114df46286b4bae10faa5738d74c6eafe33d04608e1a559292b0a6f08
Jul 15 21:15:04.672538 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:04.672522    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 15 21:15:04.769217 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:04.769181    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt" event={"ID":"0f7d9980-d5ed-4368-9f06-4eb0885b2e9d","Type":"ContainerStarted","Data":"f47b50e114df46286b4bae10faa5738d74c6eafe33d04608e1a559292b0a6f08"}
Jul 15 21:15:05.375424 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:05.375388    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:15:05.469849 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:05.469810    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:15:05.628681 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:05.628602    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8143c542-6586fdcdbc-qfzx4"]
Jul 15 21:15:05.633866 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:05.633841    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8143c542-6586fdcdbc-qfzx4"
Jul 15 21:15:05.637304 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:05.637281    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 15 21:15:05.637545 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:05.637528    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-s2l9d\""
Jul 15 21:15:05.637610 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:05.637528    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-8143c542-serving-cert\""
Jul 15 21:15:05.638691 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:05.638675    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 15 21:15:05.647047 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:05.647018    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8143c542-6586fdcdbc-qfzx4"]
Jul 15 21:15:05.670736 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:05.670700    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:15:05.683075 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:05.683042    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/201a732c-4fc5-476c-98d9-635917dd1e11-maas-api-tls\") pod \"maas-api-e2e-ait-8143c542-6586fdcdbc-qfzx4\" (UID: \"201a732c-4fc5-476c-98d9-635917dd1e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8143c542-6586fdcdbc-qfzx4"
Jul 15 21:15:05.683222 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:05.683121    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmbmw\" (UniqueName: \"kubernetes.io/projected/201a732c-4fc5-476c-98d9-635917dd1e11-kube-api-access-mmbmw\") pod \"maas-api-e2e-ait-8143c542-6586fdcdbc-qfzx4\" (UID: \"201a732c-4fc5-476c-98d9-635917dd1e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8143c542-6586fdcdbc-qfzx4"
Jul 15 21:15:05.683222 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:05.683196    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/201a732c-4fc5-476c-98d9-635917dd1e11-openshift-service-ca\") pod \"maas-api-e2e-ait-8143c542-6586fdcdbc-qfzx4\" (UID: \"201a732c-4fc5-476c-98d9-635917dd1e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8143c542-6586fdcdbc-qfzx4"
Jul 15 21:15:05.784945 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:05.784378    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/201a732c-4fc5-476c-98d9-635917dd1e11-openshift-service-ca\") pod \"maas-api-e2e-ait-8143c542-6586fdcdbc-qfzx4\" (UID: \"201a732c-4fc5-476c-98d9-635917dd1e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8143c542-6586fdcdbc-qfzx4"
Jul 15 21:15:05.784945 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:05.784491    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/201a732c-4fc5-476c-98d9-635917dd1e11-maas-api-tls\") pod \"maas-api-e2e-ait-8143c542-6586fdcdbc-qfzx4\" (UID: \"201a732c-4fc5-476c-98d9-635917dd1e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8143c542-6586fdcdbc-qfzx4"
Jul 15 21:15:05.784945 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:05.784575    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmbmw\" (UniqueName: \"kubernetes.io/projected/201a732c-4fc5-476c-98d9-635917dd1e11-kube-api-access-mmbmw\") pod \"maas-api-e2e-ait-8143c542-6586fdcdbc-qfzx4\" (UID: \"201a732c-4fc5-476c-98d9-635917dd1e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8143c542-6586fdcdbc-qfzx4"
Jul 15 21:15:05.785363 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:05.785328    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/201a732c-4fc5-476c-98d9-635917dd1e11-openshift-service-ca\") pod \"maas-api-e2e-ait-8143c542-6586fdcdbc-qfzx4\" (UID: \"201a732c-4fc5-476c-98d9-635917dd1e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8143c542-6586fdcdbc-qfzx4"
Jul 15 21:15:05.787567 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:05.787545    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/201a732c-4fc5-476c-98d9-635917dd1e11-maas-api-tls\") pod \"maas-api-e2e-ait-8143c542-6586fdcdbc-qfzx4\" (UID: \"201a732c-4fc5-476c-98d9-635917dd1e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8143c542-6586fdcdbc-qfzx4"
Jul 15 21:15:05.794097 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:05.794052    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmbmw\" (UniqueName: \"kubernetes.io/projected/201a732c-4fc5-476c-98d9-635917dd1e11-kube-api-access-mmbmw\") pod \"maas-api-e2e-ait-8143c542-6586fdcdbc-qfzx4\" (UID: \"201a732c-4fc5-476c-98d9-635917dd1e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8143c542-6586fdcdbc-qfzx4"
Jul 15 21:15:05.822169 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:05.822099    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:15:05.945779 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:05.945746    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8143c542-6586fdcdbc-qfzx4"
Jul 15 21:15:06.156501 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:06.156467    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:15:06.254102 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:06.254065    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:15:06.265238 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:06.265211    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8143c542-6586fdcdbc-qfzx4"]
Jul 15 21:15:06.268237 ip-10-0-137-234 kubenswrapper[2566]: W0715 21:15:06.268207    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod201a732c_4fc5_476c_98d9_635917dd1e11.slice/crio-d8ca0aa5e2dd9dfeba996cd395dfc4681100c08f1c96703ac4c376331a914668 WatchSource:0}: Error finding container d8ca0aa5e2dd9dfeba996cd395dfc4681100c08f1c96703ac4c376331a914668: Status 404 returned error can't find the container with id d8ca0aa5e2dd9dfeba996cd395dfc4681100c08f1c96703ac4c376331a914668
Jul 15 21:15:06.782464 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:06.782427    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8143c542-6586fdcdbc-qfzx4" event={"ID":"201a732c-4fc5-476c-98d9-635917dd1e11","Type":"ContainerStarted","Data":"d8ca0aa5e2dd9dfeba996cd395dfc4681100c08f1c96703ac4c376331a914668"}
Jul 15 21:15:07.844761 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:07.844722    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:15:07.845068 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:07.844830    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:15:07.845068 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:07.844862    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:15:08.777454 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:08.777401    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:15:08.793199 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:08.793164    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt" event={"ID":"0f7d9980-d5ed-4368-9f06-4eb0885b2e9d","Type":"ContainerStarted","Data":"aaaf95c9643d2817557fd54b18b642498c0928141d37a82d2f6c638bafc351bb"}
Jul 15 21:15:08.817619 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:08.817566    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt" podStartSLOduration=1.645761406 podStartE2EDuration="4.817547751s" totalImagesPullingTime="3.171786345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:15:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:15:08.816055212 +0000 UTC m=+1641.965093379" watchObservedRunningTime="2026-07-15 21:15:08.817547751 +0000 UTC m=+1641.966585809"
Jul 15 21:15:09.531219 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:09.531177    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt"
Jul 15 21:15:09.535732 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:09.535712    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt"
Jul 15 21:15:09.799696 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:09.799662    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8143c542-6586fdcdbc-qfzx4" event={"ID":"201a732c-4fc5-476c-98d9-635917dd1e11","Type":"ContainerStarted","Data":"b4841e2fcadd6c293117d0ce96d5fbfbec4ff4b1754f4237ff0dafbb35579467"}
Jul 15 21:15:09.799917 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:09.799755    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8143c542-6586fdcdbc-qfzx4"
Jul 15 21:15:09.799917 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:09.799801    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt"
Jul 15 21:15:09.800991 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:09.800972    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt"
Jul 15 21:15:09.817822 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:09.817569    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8143c542-6586fdcdbc-qfzx4" podStartSLOduration=1.790896829 podStartE2EDuration="4.817544634s" totalImagesPullingTime="3.026647805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:15:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:15:09.81727735 +0000 UTC m=+1642.966315426" watchObservedRunningTime="2026-07-15 21:15:09.817544634 +0000 UTC m=+1642.966582691"
Jul 15 21:15:10.443010 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:10.442982    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8143c542-s9grj"]
Jul 15 21:15:10.447026 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:10.447001    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8143c542-s9grj"
Jul 15 21:15:10.450360 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:10.450331    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zsrf5\""
Jul 15 21:15:10.455551 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:10.455519    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8143c542-s9grj"]
Jul 15 21:15:10.531939 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:10.531875    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ffe389c-1dce-44d4-87ca-21c4c9e98d5d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8143c542-s9grj\" (UID: \"2ffe389c-1dce-44d4-87ca-21c4c9e98d5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8143c542-s9grj"
Jul 15 21:15:10.632916 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:10.632849    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ffe389c-1dce-44d4-87ca-21c4c9e98d5d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8143c542-s9grj\" (UID: \"2ffe389c-1dce-44d4-87ca-21c4c9e98d5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8143c542-s9grj"
Jul 15 21:15:10.633526 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:10.633501    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ffe389c-1dce-44d4-87ca-21c4c9e98d5d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8143c542-s9grj\" (UID: \"2ffe389c-1dce-44d4-87ca-21c4c9e98d5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8143c542-s9grj"
Jul 15 21:15:10.758233 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:10.758134    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8143c542-s9grj"
Jul 15 21:15:10.955546 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:10.953237    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8143c542-s9grj"]
Jul 15 21:15:10.958642 ip-10-0-137-234 kubenswrapper[2566]: W0715 21:15:10.958616    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ffe389c_1dce_44d4_87ca_21c4c9e98d5d.slice/crio-a78e8350b1012f18c89701752da71a43922ed0e9fe93b087300289cefbe513fd WatchSource:0}: Error finding container a78e8350b1012f18c89701752da71a43922ed0e9fe93b087300289cefbe513fd: Status 404 returned error can't find the container with id a78e8350b1012f18c89701752da71a43922ed0e9fe93b087300289cefbe513fd
Jul 15 21:15:11.817195 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:11.817158    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8143c542-s9grj" event={"ID":"2ffe389c-1dce-44d4-87ca-21c4c9e98d5d","Type":"ContainerStarted","Data":"d74a3eef757cbccc06d77b296e8d2e96cd27af1dba61af7e9b9ceafefda1f020"}
Jul 15 21:15:11.817571 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:11.817218    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8143c542-s9grj" event={"ID":"2ffe389c-1dce-44d4-87ca-21c4c9e98d5d","Type":"ContainerStarted","Data":"a78e8350b1012f18c89701752da71a43922ed0e9fe93b087300289cefbe513fd"}
Jul 15 21:15:11.838308 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:11.838262    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8143c542-s9grj" podStartSLOduration=1.838245898 podStartE2EDuration="1.838245898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:15:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:15:11.837088865 +0000 UTC m=+1644.986126934" watchObservedRunningTime="2026-07-15 21:15:11.838245898 +0000 UTC m=+1644.987283955"
Jul 15 21:15:12.823549 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:12.823514    2566 generic.go:356] "Generic (PLEG): container finished" podID="2ffe389c-1dce-44d4-87ca-21c4c9e98d5d" containerID="d74a3eef757cbccc06d77b296e8d2e96cd27af1dba61af7e9b9ceafefda1f020" exitCode=7
Jul 15 21:15:12.824013 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:12.823598    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8143c542-s9grj" event={"ID":"2ffe389c-1dce-44d4-87ca-21c4c9e98d5d","Type":"ContainerDied","Data":"d74a3eef757cbccc06d77b296e8d2e96cd27af1dba61af7e9b9ceafefda1f020"}
Jul 15 21:15:12.824013 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:12.823760    2566 scope.go:117] "RemoveContainer" containerID="d74a3eef757cbccc06d77b296e8d2e96cd27af1dba61af7e9b9ceafefda1f020"
Jul 15 21:15:13.830219 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:13.830186    2566 generic.go:356] "Generic (PLEG): container finished" podID="2ffe389c-1dce-44d4-87ca-21c4c9e98d5d" containerID="7b990219a6396c75e835c6f92b13503eb427f152a8bc31a89e7f8b85c1d0bd3e" exitCode=7
Jul 15 21:15:13.830663 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:13.830270    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8143c542-s9grj" event={"ID":"2ffe389c-1dce-44d4-87ca-21c4c9e98d5d","Type":"ContainerDied","Data":"7b990219a6396c75e835c6f92b13503eb427f152a8bc31a89e7f8b85c1d0bd3e"}
Jul 15 21:15:13.830663 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:13.830335    2566 scope.go:117] "RemoveContainer" containerID="d74a3eef757cbccc06d77b296e8d2e96cd27af1dba61af7e9b9ceafefda1f020"
Jul 15 21:15:13.830663 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:13.830457    2566 scope.go:117] "RemoveContainer" containerID="7b990219a6396c75e835c6f92b13503eb427f152a8bc31a89e7f8b85c1d0bd3e"
Jul 15 21:15:13.830789 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:15:13.830669    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8143c542-s9grj_odh-ai-gateway-infra(2ffe389c-1dce-44d4-87ca-21c4c9e98d5d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8143c542-s9grj" podUID="2ffe389c-1dce-44d4-87ca-21c4c9e98d5d"
Jul 15 21:15:14.836440 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:14.836409    2566 scope.go:117] "RemoveContainer" containerID="7b990219a6396c75e835c6f92b13503eb427f152a8bc31a89e7f8b85c1d0bd3e"
Jul 15 21:15:14.836962 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:15:14.836701    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8143c542-s9grj_odh-ai-gateway-infra(2ffe389c-1dce-44d4-87ca-21c4c9e98d5d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8143c542-s9grj" podUID="2ffe389c-1dce-44d4-87ca-21c4c9e98d5d"
Jul 15 21:15:15.817745 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:15.817714    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8143c542-6586fdcdbc-qfzx4"
Jul 15 21:15:17.851872 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:17.851837    2566 generic.go:356] "Generic (PLEG): container finished" podID="0f7d9980-d5ed-4368-9f06-4eb0885b2e9d" containerID="aaaf95c9643d2817557fd54b18b642498c0928141d37a82d2f6c638bafc351bb" exitCode=0
Jul 15 21:15:17.852462 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:17.851915    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt" event={"ID":"0f7d9980-d5ed-4368-9f06-4eb0885b2e9d","Type":"ContainerDied","Data":"aaaf95c9643d2817557fd54b18b642498c0928141d37a82d2f6c638bafc351bb"}
Jul 15 21:15:17.852462 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:17.852281    2566 scope.go:117] "RemoveContainer" containerID="aaaf95c9643d2817557fd54b18b642498c0928141d37a82d2f6c638bafc351bb"
Jul 15 21:15:17.854700 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:17.854664    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:15:17.854829 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:17.854737    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:15:17.854829 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:17.854770    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:15:18.857786 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:18.857746    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt" event={"ID":"0f7d9980-d5ed-4368-9f06-4eb0885b2e9d","Type":"ContainerStarted","Data":"92a94f400f8b69de3ccc0c300d6ff54b729f7e4e0b2d09403e54d99ef5ea3aee"}
Jul 15 21:15:19.531445 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:19.531409    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt"
Jul 15 21:15:19.531701 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:19.531679    2566 patch_prober.go:28] interesting pod/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Jul 15 21:15:19.531748 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:19.531727    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt" podUID="0f7d9980-d5ed-4368-9f06-4eb0885b2e9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Jul 15 21:15:20.531574 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:20.531533    2566 patch_prober.go:28] interesting pod/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Jul 15 21:15:20.531994 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:20.531592    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt" podUID="0f7d9980-d5ed-4368-9f06-4eb0885b2e9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Jul 15 21:15:21.531378 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:21.531331    2566 patch_prober.go:28] interesting pod/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Jul 15 21:15:21.531543 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:21.531389    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt" podUID="0f7d9980-d5ed-4368-9f06-4eb0885b2e9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Jul 15 21:15:22.531482 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:22.531442    2566 patch_prober.go:28] interesting pod/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Jul 15 21:15:22.531930 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:22.531500    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt" podUID="0f7d9980-d5ed-4368-9f06-4eb0885b2e9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Jul 15 21:15:23.535672 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:23.535641    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt"
Jul 15 21:15:23.536070 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:23.535919    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt"
Jul 15 21:15:23.536662 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:23.536639    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt"
Jul 15 21:15:29.751335 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:29.751292    2566 scope.go:117] "RemoveContainer" containerID="7b990219a6396c75e835c6f92b13503eb427f152a8bc31a89e7f8b85c1d0bd3e"
Jul 15 21:15:30.913431 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:30.913347    2566 generic.go:356] "Generic (PLEG): container finished" podID="2ffe389c-1dce-44d4-87ca-21c4c9e98d5d" containerID="96b9db41f52f295275ba34df02df72eea04e3559342973fd554699b0e3e41fcb" exitCode=0
Jul 15 21:15:30.913431 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:30.913397    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8143c542-s9grj" event={"ID":"2ffe389c-1dce-44d4-87ca-21c4c9e98d5d","Type":"ContainerDied","Data":"96b9db41f52f295275ba34df02df72eea04e3559342973fd554699b0e3e41fcb"}
Jul 15 21:15:30.913929 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:30.913442    2566 scope.go:117] "RemoveContainer" containerID="7b990219a6396c75e835c6f92b13503eb427f152a8bc31a89e7f8b85c1d0bd3e"
Jul 15 21:15:31.945986 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:31.945956    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8143c542-s9grj"]
Jul 15 21:15:32.046003 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:32.045981    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8143c542-s9grj"
Jul 15 21:15:32.230177 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:32.230143    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ffe389c-1dce-44d4-87ca-21c4c9e98d5d-maas-api-service-ca\") pod \"2ffe389c-1dce-44d4-87ca-21c4c9e98d5d\" (UID: \"2ffe389c-1dce-44d4-87ca-21c4c9e98d5d\") "
Jul 15 21:15:32.230521 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:32.230497    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ffe389c-1dce-44d4-87ca-21c4c9e98d5d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2ffe389c-1dce-44d4-87ca-21c4c9e98d5d" (UID: "2ffe389c-1dce-44d4-87ca-21c4c9e98d5d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 21:15:32.331672 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:32.331622    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ffe389c-1dce-44d4-87ca-21c4c9e98d5d-maas-api-service-ca\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:15:32.925459 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:32.925423    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8143c542-s9grj"
Jul 15 21:15:32.925459 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:32.925443    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8143c542-s9grj" event={"ID":"2ffe389c-1dce-44d4-87ca-21c4c9e98d5d","Type":"ContainerDied","Data":"a78e8350b1012f18c89701752da71a43922ed0e9fe93b087300289cefbe513fd"}
Jul 15 21:15:32.925779 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:32.925484    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a78e8350b1012f18c89701752da71a43922ed0e9fe93b087300289cefbe513fd"
Jul 15 21:15:32.957328 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:32.957296    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8143c542-s9grj"]
Jul 15 21:15:32.963707 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:32.963684    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8143c542-s9grj"]
Jul 15 21:15:33.757045 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:15:33.757015    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ffe389c-1dce-44d4-87ca-21c4c9e98d5d" path="/var/lib/kubelet/pods/2ffe389c-1dce-44d4-87ca-21c4c9e98d5d/volumes"
Jul 15 21:16:10.504460 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:10.504420    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8143c542-cxpf6"]
Jul 15 21:16:10.504880 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:10.504778    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ffe389c-1dce-44d4-87ca-21c4c9e98d5d" containerName="revoke-keys"
Jul 15 21:16:10.504880 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:10.504790    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ffe389c-1dce-44d4-87ca-21c4c9e98d5d" containerName="revoke-keys"
Jul 15 21:16:10.504880 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:10.504798    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ffe389c-1dce-44d4-87ca-21c4c9e98d5d" containerName="revoke-keys"
Jul 15 21:16:10.504880 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:10.504805    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ffe389c-1dce-44d4-87ca-21c4c9e98d5d" containerName="revoke-keys"
Jul 15 21:16:10.504880 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:10.504860    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ffe389c-1dce-44d4-87ca-21c4c9e98d5d" containerName="revoke-keys"
Jul 15 21:16:10.504880 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:10.504867    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ffe389c-1dce-44d4-87ca-21c4c9e98d5d" containerName="revoke-keys"
Jul 15 21:16:10.504880 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:10.504874    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ffe389c-1dce-44d4-87ca-21c4c9e98d5d" containerName="revoke-keys"
Jul 15 21:16:10.506969 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:10.506952    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8143c542-cxpf6"
Jul 15 21:16:10.509879 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:10.509859    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zsrf5\""
Jul 15 21:16:10.516593 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:10.516565    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8143c542-cxpf6"]
Jul 15 21:16:10.657921 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:10.657868    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1b4397d-0ffa-4c58-af9d-5dd79f8c79c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8143c542-cxpf6\" (UID: \"e1b4397d-0ffa-4c58-af9d-5dd79f8c79c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8143c542-cxpf6"
Jul 15 21:16:10.758543 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:10.758460    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1b4397d-0ffa-4c58-af9d-5dd79f8c79c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8143c542-cxpf6\" (UID: \"e1b4397d-0ffa-4c58-af9d-5dd79f8c79c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8143c542-cxpf6"
Jul 15 21:16:10.759128 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:10.759108    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1b4397d-0ffa-4c58-af9d-5dd79f8c79c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8143c542-cxpf6\" (UID: \"e1b4397d-0ffa-4c58-af9d-5dd79f8c79c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8143c542-cxpf6"
Jul 15 21:16:10.817738 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:10.817699    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8143c542-cxpf6"
Jul 15 21:16:10.952804 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:10.952759    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8143c542-cxpf6"]
Jul 15 21:16:10.955220 ip-10-0-137-234 kubenswrapper[2566]: W0715 21:16:10.955180    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1b4397d_0ffa_4c58_af9d_5dd79f8c79c1.slice/crio-e555d50299697855fabbbc2a61496cda0e010b61e0ad9a4c811247d288718e7c WatchSource:0}: Error finding container e555d50299697855fabbbc2a61496cda0e010b61e0ad9a4c811247d288718e7c: Status 404 returned error can't find the container with id e555d50299697855fabbbc2a61496cda0e010b61e0ad9a4c811247d288718e7c
Jul 15 21:16:11.095790 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:11.095705    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8143c542-cxpf6" event={"ID":"e1b4397d-0ffa-4c58-af9d-5dd79f8c79c1","Type":"ContainerStarted","Data":"e555d50299697855fabbbc2a61496cda0e010b61e0ad9a4c811247d288718e7c"}
Jul 15 21:16:12.100945 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:12.100910    2566 generic.go:356] "Generic (PLEG): container finished" podID="e1b4397d-0ffa-4c58-af9d-5dd79f8c79c1" containerID="a0fe05f7f1ab790123dcb646a9abdac0c345901462181b8acc9ae1c1af41b17d" exitCode=0
Jul 15 21:16:12.101368 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:12.100996    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8143c542-cxpf6" event={"ID":"e1b4397d-0ffa-4c58-af9d-5dd79f8c79c1","Type":"ContainerDied","Data":"a0fe05f7f1ab790123dcb646a9abdac0c345901462181b8acc9ae1c1af41b17d"}
Jul 15 21:16:13.235107 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:13.235082    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8143c542-cxpf6"
Jul 15 21:16:13.286059 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:13.286025    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:16:13.353505 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:13.353478    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:16:13.380860 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:13.380781    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1b4397d-0ffa-4c58-af9d-5dd79f8c79c1-maas-api-service-ca\") pod \"e1b4397d-0ffa-4c58-af9d-5dd79f8c79c1\" (UID: \"e1b4397d-0ffa-4c58-af9d-5dd79f8c79c1\") "
Jul 15 21:16:13.381164 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:13.381141    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1b4397d-0ffa-4c58-af9d-5dd79f8c79c1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e1b4397d-0ffa-4c58-af9d-5dd79f8c79c1" (UID: "e1b4397d-0ffa-4c58-af9d-5dd79f8c79c1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 21:16:13.482243 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:13.482215    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1b4397d-0ffa-4c58-af9d-5dd79f8c79c1-maas-api-service-ca\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:16:14.110927 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:14.110821    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8143c542-cxpf6" event={"ID":"e1b4397d-0ffa-4c58-af9d-5dd79f8c79c1","Type":"ContainerDied","Data":"e555d50299697855fabbbc2a61496cda0e010b61e0ad9a4c811247d288718e7c"}
Jul 15 21:16:14.110927 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:14.110861    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8143c542-cxpf6"
Jul 15 21:16:14.110927 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:14.110877    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e555d50299697855fabbbc2a61496cda0e010b61e0ad9a4c811247d288718e7c"
Jul 15 21:16:21.173869 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:21.173829    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8143c542-6586fdcdbc-qfzx4"]
Jul 15 21:16:21.174406 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:21.174129    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8143c542-6586fdcdbc-qfzx4" podUID="201a732c-4fc5-476c-98d9-635917dd1e11" containerName="maas-api" containerID="cri-o://b4841e2fcadd6c293117d0ce96d5fbfbec4ff4b1754f4237ff0dafbb35579467" gracePeriod=30
Jul 15 21:16:21.223953 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:21.223918    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:16:21.423597 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:21.423574    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8143c542-6586fdcdbc-qfzx4"
Jul 15 21:16:21.555536 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:21.555498    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/201a732c-4fc5-476c-98d9-635917dd1e11-maas-api-tls\") pod \"201a732c-4fc5-476c-98d9-635917dd1e11\" (UID: \"201a732c-4fc5-476c-98d9-635917dd1e11\") "
Jul 15 21:16:21.555739 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:21.555584    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/201a732c-4fc5-476c-98d9-635917dd1e11-openshift-service-ca\") pod \"201a732c-4fc5-476c-98d9-635917dd1e11\" (UID: \"201a732c-4fc5-476c-98d9-635917dd1e11\") "
Jul 15 21:16:21.555739 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:21.555609    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmbmw\" (UniqueName: \"kubernetes.io/projected/201a732c-4fc5-476c-98d9-635917dd1e11-kube-api-access-mmbmw\") pod \"201a732c-4fc5-476c-98d9-635917dd1e11\" (UID: \"201a732c-4fc5-476c-98d9-635917dd1e11\") "
Jul 15 21:16:21.556026 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:21.556003    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/201a732c-4fc5-476c-98d9-635917dd1e11-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "201a732c-4fc5-476c-98d9-635917dd1e11" (UID: "201a732c-4fc5-476c-98d9-635917dd1e11"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 21:16:21.557965 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:21.557866    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/201a732c-4fc5-476c-98d9-635917dd1e11-kube-api-access-mmbmw" (OuterVolumeSpecName: "kube-api-access-mmbmw") pod "201a732c-4fc5-476c-98d9-635917dd1e11" (UID: "201a732c-4fc5-476c-98d9-635917dd1e11"). InnerVolumeSpecName "kube-api-access-mmbmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 21:16:21.557965 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:21.557905    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/201a732c-4fc5-476c-98d9-635917dd1e11-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "201a732c-4fc5-476c-98d9-635917dd1e11" (UID: "201a732c-4fc5-476c-98d9-635917dd1e11"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 21:16:21.656281 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:21.656246    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/201a732c-4fc5-476c-98d9-635917dd1e11-openshift-service-ca\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:16:21.656281 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:21.656274    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmbmw\" (UniqueName: \"kubernetes.io/projected/201a732c-4fc5-476c-98d9-635917dd1e11-kube-api-access-mmbmw\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:16:21.656281 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:21.656285    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/201a732c-4fc5-476c-98d9-635917dd1e11-maas-api-tls\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:16:22.146797 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:22.146762    2566 generic.go:356] "Generic (PLEG): container finished" podID="201a732c-4fc5-476c-98d9-635917dd1e11" containerID="b4841e2fcadd6c293117d0ce96d5fbfbec4ff4b1754f4237ff0dafbb35579467" exitCode=0
Jul 15 21:16:22.147022 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:22.146813    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8143c542-6586fdcdbc-qfzx4" event={"ID":"201a732c-4fc5-476c-98d9-635917dd1e11","Type":"ContainerDied","Data":"b4841e2fcadd6c293117d0ce96d5fbfbec4ff4b1754f4237ff0dafbb35579467"}
Jul 15 21:16:22.147022 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:22.146861    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8143c542-6586fdcdbc-qfzx4" event={"ID":"201a732c-4fc5-476c-98d9-635917dd1e11","Type":"ContainerDied","Data":"d8ca0aa5e2dd9dfeba996cd395dfc4681100c08f1c96703ac4c376331a914668"}
Jul 15 21:16:22.147022 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:22.146815    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8143c542-6586fdcdbc-qfzx4"
Jul 15 21:16:22.147022 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:22.146915    2566 scope.go:117] "RemoveContainer" containerID="b4841e2fcadd6c293117d0ce96d5fbfbec4ff4b1754f4237ff0dafbb35579467"
Jul 15 21:16:22.155396 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:22.155377    2566 scope.go:117] "RemoveContainer" containerID="b4841e2fcadd6c293117d0ce96d5fbfbec4ff4b1754f4237ff0dafbb35579467"
Jul 15 21:16:22.155625 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:16:22.155607    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4841e2fcadd6c293117d0ce96d5fbfbec4ff4b1754f4237ff0dafbb35579467\": container with ID starting with b4841e2fcadd6c293117d0ce96d5fbfbec4ff4b1754f4237ff0dafbb35579467 not found: ID does not exist" containerID="b4841e2fcadd6c293117d0ce96d5fbfbec4ff4b1754f4237ff0dafbb35579467"
Jul 15 21:16:22.155676 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:22.155634    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4841e2fcadd6c293117d0ce96d5fbfbec4ff4b1754f4237ff0dafbb35579467"} err="failed to get container status \"b4841e2fcadd6c293117d0ce96d5fbfbec4ff4b1754f4237ff0dafbb35579467\": rpc error: code = NotFound desc = could not find container \"b4841e2fcadd6c293117d0ce96d5fbfbec4ff4b1754f4237ff0dafbb35579467\": container with ID starting with b4841e2fcadd6c293117d0ce96d5fbfbec4ff4b1754f4237ff0dafbb35579467 not found: ID does not exist"
Jul 15 21:16:22.167608 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:22.167583    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8143c542-6586fdcdbc-qfzx4"]
Jul 15 21:16:22.172030 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:22.172011    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8143c542-6586fdcdbc-qfzx4"]
Jul 15 21:16:23.756455 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:23.756423    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="201a732c-4fc5-476c-98d9-635917dd1e11" path="/var/lib/kubelet/pods/201a732c-4fc5-476c-98d9-635917dd1e11/volumes"
Jul 15 21:16:25.606381 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:25.606311    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:16:32.385814 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:32.385773    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:16:32.499345 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:32.499308    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:16:35.109468 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:35.109422    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:16:35.808705 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:35.808665    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt"]
Jul 15 21:16:35.808980 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:35.808940    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt" podUID="0f7d9980-d5ed-4368-9f06-4eb0885b2e9d" containerName="istio-proxy" containerID="cri-o://92a94f400f8b69de3ccc0c300d6ff54b729f7e4e0b2d09403e54d99ef5ea3aee" gracePeriod=30
Jul 15 21:16:35.832113 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:35.832080    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:16:36.232439 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:36.232408    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:16:36.345553 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:36.345519    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:16:36.411477 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:36.411448    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7"]
Jul 15 21:16:36.411852 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:36.411837    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1b4397d-0ffa-4c58-af9d-5dd79f8c79c1" containerName="revoke-keys"
Jul 15 21:16:36.411852 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:36.411853    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1b4397d-0ffa-4c58-af9d-5dd79f8c79c1" containerName="revoke-keys"
Jul 15 21:16:36.411984 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:36.411867    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="201a732c-4fc5-476c-98d9-635917dd1e11" containerName="maas-api"
Jul 15 21:16:36.411984 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:36.411873    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="201a732c-4fc5-476c-98d9-635917dd1e11" containerName="maas-api"
Jul 15 21:16:36.411984 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:36.411881    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ffe389c-1dce-44d4-87ca-21c4c9e98d5d" containerName="revoke-keys"
Jul 15 21:16:36.411984 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:36.411886    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ffe389c-1dce-44d4-87ca-21c4c9e98d5d" containerName="revoke-keys"
Jul 15 21:16:36.411984 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:36.411961    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1b4397d-0ffa-4c58-af9d-5dd79f8c79c1" containerName="revoke-keys"
Jul 15 21:16:36.411984 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:36.411973    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="201a732c-4fc5-476c-98d9-635917dd1e11" containerName="maas-api"
Jul 15 21:16:36.416243 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:36.416221    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7"
Jul 15 21:16:36.420182 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:36.420161    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-baea565a-gw-openshift-default-dockercfg-ft8mf\""
Jul 15 21:16:36.439908 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:36.439874    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7"]
Jul 15 21:16:36.475222 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:36.475195    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:16:36.487524 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:36.487450    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc0cedd8-e570-4a06-b010-a1603c763de3-workload-socket\") pod \"e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7\" (UID: \"dc0cedd8-e570-4a06-b010-a1603c763de3\") " pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7"
Jul 15 21:16:36.487524 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:36.487494    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc0cedd8-e570-4a06-b010-a1603c763de3-istio-envoy\") pod \"e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7\" (UID: \"dc0cedd8-e570-4a06-b010-a1603c763de3\") " pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7"
Jul 15 21:16:36.487674 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:36.487547    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc0cedd8-e570-4a06-b010-a1603c763de3-istiod-ca-cert\") pod \"e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7\" (UID: \"dc0cedd8-e570-4a06-b010-a1603c763de3\") " pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7"
Jul 15 21:16:36.487674 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:36.487579    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc0cedd8-e570-4a06-b010-a1603c763de3-istio-data\") pod \"e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7\" (UID: \"dc0cedd8-e570-4a06-b010-a1603c763de3\") " pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7"
Jul 15 21:16:36.487674 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:36.487654    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc0cedd8-e570-4a06-b010-a1603c763de3-workload-certs\") pod \"e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7\" (UID: \"dc0cedd8-e570-4a06-b010-a1603c763de3\") " pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7"
Jul 15 21:16:36.487786 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:36.487683    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc0cedd8-e570-4a06-b010-a1603c763de3-istio-token\") pod \"e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7\" (UID: \"dc0cedd8-e570-4a06-b010-a1603c763de3\") " pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7"
Jul 15 21:16:36.487786 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:36.487737    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv8ft\" (UniqueName: \"kubernetes.io/projected/dc0cedd8-e570-4a06-b010-a1603c763de3-kube-api-access-vv8ft\") pod \"e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7\" (UID: \"dc0cedd8-e570-4a06-b010-a1603c763de3\") " pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7"
Jul 15 21:16:36.487861 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:36.487795    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc0cedd8-e570-4a06-b010-a1603c763de3-istio-podinfo\") pod \"e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7\" (UID: \"dc0cedd8-e570-4a06-b010-a1603c763de3\") " pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7"
Jul 15 21:16:36.487861 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:36.487829    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc0cedd8-e570-4a06-b010-a1603c763de3-credential-socket\") pod \"e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7\" (UID: \"dc0cedd8-e570-4a06-b010-a1603c763de3\") " pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7"
Jul 15 21:16:36.588504 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:36.588469    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc0cedd8-e570-4a06-b010-a1603c763de3-workload-certs\") pod \"e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7\" (UID: \"dc0cedd8-e570-4a06-b010-a1603c763de3\") " pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7"
Jul 15 21:16:36.588504 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:36.588505    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc0cedd8-e570-4a06-b010-a1603c763de3-istio-token\") pod \"e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7\" (UID: \"dc0cedd8-e570-4a06-b010-a1603c763de3\") " pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7"
Jul 15 21:16:36.588711 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:36.588554    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv8ft\" (UniqueName: \"kubernetes.io/projected/dc0cedd8-e570-4a06-b010-a1603c763de3-kube-api-access-vv8ft\") pod \"e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7\" (UID: \"dc0cedd8-e570-4a06-b010-a1603c763de3\") " pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7"
Jul 15 21:16:36.588711 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:36.588579    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc0cedd8-e570-4a06-b010-a1603c763de3-istio-podinfo\") pod \"e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7\" (UID: \"dc0cedd8-e570-4a06-b010-a1603c763de3\") " pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7"
Jul 15 21:16:36.588711 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:36.588613    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc0cedd8-e570-4a06-b010-a1603c763de3-credential-socket\") pod \"e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7\" (UID: \"dc0cedd8-e570-4a06-b010-a1603c763de3\") " pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7"
Jul 15 21:16:36.588711 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:36.588654    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc0cedd8-e570-4a06-b010-a1603c763de3-workload-socket\") pod \"e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7\" (UID: \"dc0cedd8-e570-4a06-b010-a1603c763de3\") " pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7"
Jul 15 21:16:36.588711 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:36.588676    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc0cedd8-e570-4a06-b010-a1603c763de3-istio-envoy\") pod \"e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7\" (UID: \"dc0cedd8-e570-4a06-b010-a1603c763de3\") " pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7"
Jul 15 21:16:36.588711 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:36.588708    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc0cedd8-e570-4a06-b010-a1603c763de3-istiod-ca-cert\") pod \"e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7\" (UID: \"dc0cedd8-e570-4a06-b010-a1603c763de3\") " pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7"
Jul 15 21:16:36.589098 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:36.588747    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc0cedd8-e570-4a06-b010-a1603c763de3-istio-data\") pod \"e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7\" (UID: \"dc0cedd8-e570-4a06-b010-a1603c763de3\") " pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7"
Jul 15 21:16:36.589098 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:36.588958    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc0cedd8-e570-4a06-b010-a1603c763de3-workload-certs\") pod \"e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7\" (UID: \"dc0cedd8-e570-4a06-b010-a1603c763de3\") " pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7"
Jul 15 21:16:36.589098 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:36.589063    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc0cedd8-e570-4a06-b010-a1603c763de3-workload-socket\") pod \"e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7\" (UID: \"dc0cedd8-e570-4a06-b010-a1603c763de3\") " pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7"
Jul 15 21:16:36.589098 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:36.589078    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc0cedd8-e570-4a06-b010-a1603c763de3-istio-data\") pod \"e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7\" (UID: \"dc0cedd8-e570-4a06-b010-a1603c763de3\") " pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7"
Jul 15 21:16:36.589281 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:36.589214    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc0cedd8-e570-4a06-b010-a1603c763de3-credential-socket\") pod \"e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7\" (UID: \"dc0cedd8-e570-4a06-b010-a1603c763de3\") " pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7"
Jul 15 21:16:36.589441 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:36.589424    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc0cedd8-e570-4a06-b010-a1603c763de3-istiod-ca-cert\") pod \"e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7\" (UID: \"dc0cedd8-e570-4a06-b010-a1603c763de3\") " pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7"
Jul 15 21:16:36.591035 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:36.591006    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc0cedd8-e570-4a06-b010-a1603c763de3-istio-envoy\") pod \"e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7\" (UID: \"dc0cedd8-e570-4a06-b010-a1603c763de3\") " pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7"
Jul 15 21:16:36.591151 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:36.591093    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc0cedd8-e570-4a06-b010-a1603c763de3-istio-podinfo\") pod \"e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7\" (UID: \"dc0cedd8-e570-4a06-b010-a1603c763de3\") " pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7"
Jul 15 21:16:36.599640 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:36.599609    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc0cedd8-e570-4a06-b010-a1603c763de3-istio-token\") pod \"e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7\" (UID: \"dc0cedd8-e570-4a06-b010-a1603c763de3\") " pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7"
Jul 15 21:16:36.600250 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:36.600232    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv8ft\" (UniqueName: \"kubernetes.io/projected/dc0cedd8-e570-4a06-b010-a1603c763de3-kube-api-access-vv8ft\") pod \"e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7\" (UID: \"dc0cedd8-e570-4a06-b010-a1603c763de3\") " pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7"
Jul 15 21:16:36.727172 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:36.727131    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7"
Jul 15 21:16:36.879075 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:36.879041    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7"]
Jul 15 21:16:36.880573 ip-10-0-137-234 kubenswrapper[2566]: W0715 21:16:36.880541    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc0cedd8_e570_4a06_b010_a1603c763de3.slice/crio-44ffa33c807d7e7a3d92509eaa087f246f8fbd9d637eff38e02901ddb95592ba WatchSource:0}: Error finding container 44ffa33c807d7e7a3d92509eaa087f246f8fbd9d637eff38e02901ddb95592ba: Status 404 returned error can't find the container with id 44ffa33c807d7e7a3d92509eaa087f246f8fbd9d637eff38e02901ddb95592ba
Jul 15 21:16:36.883193 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:36.883159    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:16:36.883277 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:36.883229    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:16:36.883277 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:36.883256    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:16:37.217530 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:37.217486    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7" event={"ID":"dc0cedd8-e570-4a06-b010-a1603c763de3","Type":"ContainerStarted","Data":"afc8cb0d7755dc3f9992a07144b576ddd231f498cb59ad2a2cf4cb97a9e24e39"}
Jul 15 21:16:37.217714 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:37.217555    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7" event={"ID":"dc0cedd8-e570-4a06-b010-a1603c763de3","Type":"ContainerStarted","Data":"44ffa33c807d7e7a3d92509eaa087f246f8fbd9d637eff38e02901ddb95592ba"}
Jul 15 21:16:37.242263 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:37.242209    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7" podStartSLOduration=1.242190665 podStartE2EDuration="1.242190665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:16:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:16:37.240482201 +0000 UTC m=+1730.389520290" watchObservedRunningTime="2026-07-15 21:16:37.242190665 +0000 UTC m=+1730.391228787"
Jul 15 21:16:37.727810 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:37.727775    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7"
Jul 15 21:16:37.732968 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:37.732941    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7"
Jul 15 21:16:37.764537 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:37.764508    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:16:37.854451 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:37.854419    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-baea565a-7764cdfdcb-26tfv"]
Jul 15 21:16:37.858254 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:37.858232    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-baea565a-7764cdfdcb-26tfv"
Jul 15 21:16:37.862146 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:37.862123    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-s2l9d\""
Jul 15 21:16:37.862290 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:37.862151    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 15 21:16:37.862290 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:37.862121    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-baea565a-serving-cert\""
Jul 15 21:16:37.862401 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:37.862313    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 15 21:16:37.870533 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:37.870487    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-baea565a-7764cdfdcb-26tfv"]
Jul 15 21:16:37.890010 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:37.889966    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:16:37.903747 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:37.903719    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bcdc6d6-3433-424b-b134-9a63e3df4bb4-openshift-service-ca\") pod \"maas-api-e2e-migrate-baea565a-7764cdfdcb-26tfv\" (UID: \"2bcdc6d6-3433-424b-b134-9a63e3df4bb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-baea565a-7764cdfdcb-26tfv"
Jul 15 21:16:37.903858 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:37.903769    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp7kl\" (UniqueName: \"kubernetes.io/projected/2bcdc6d6-3433-424b-b134-9a63e3df4bb4-kube-api-access-vp7kl\") pod \"maas-api-e2e-migrate-baea565a-7764cdfdcb-26tfv\" (UID: \"2bcdc6d6-3433-424b-b134-9a63e3df4bb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-baea565a-7764cdfdcb-26tfv"
Jul 15 21:16:37.903920 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:37.903856    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bcdc6d6-3433-424b-b134-9a63e3df4bb4-maas-api-tls\") pod \"maas-api-e2e-migrate-baea565a-7764cdfdcb-26tfv\" (UID: \"2bcdc6d6-3433-424b-b134-9a63e3df4bb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-baea565a-7764cdfdcb-26tfv"
Jul 15 21:16:38.004807 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:38.004713    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bcdc6d6-3433-424b-b134-9a63e3df4bb4-maas-api-tls\") pod \"maas-api-e2e-migrate-baea565a-7764cdfdcb-26tfv\" (UID: \"2bcdc6d6-3433-424b-b134-9a63e3df4bb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-baea565a-7764cdfdcb-26tfv"
Jul 15 21:16:38.004807 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:38.004793    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bcdc6d6-3433-424b-b134-9a63e3df4bb4-openshift-service-ca\") pod \"maas-api-e2e-migrate-baea565a-7764cdfdcb-26tfv\" (UID: \"2bcdc6d6-3433-424b-b134-9a63e3df4bb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-baea565a-7764cdfdcb-26tfv"
Jul 15 21:16:38.005074 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:38.004837    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp7kl\" (UniqueName: \"kubernetes.io/projected/2bcdc6d6-3433-424b-b134-9a63e3df4bb4-kube-api-access-vp7kl\") pod \"maas-api-e2e-migrate-baea565a-7764cdfdcb-26tfv\" (UID: \"2bcdc6d6-3433-424b-b134-9a63e3df4bb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-baea565a-7764cdfdcb-26tfv"
Jul 15 21:16:38.005074 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:16:38.004848    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-baea565a-serving-cert: secret "maas-api-e2e-migrate-baea565a-serving-cert" not found
Jul 15 21:16:38.005074 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:16:38.004952    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bcdc6d6-3433-424b-b134-9a63e3df4bb4-maas-api-tls podName:2bcdc6d6-3433-424b-b134-9a63e3df4bb4 nodeName:}" failed. No retries permitted until 2026-07-15 21:16:38.504932791 +0000 UTC m=+1731.653970845 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2bcdc6d6-3433-424b-b134-9a63e3df4bb4-maas-api-tls") pod "maas-api-e2e-migrate-baea565a-7764cdfdcb-26tfv" (UID: "2bcdc6d6-3433-424b-b134-9a63e3df4bb4") : secret "maas-api-e2e-migrate-baea565a-serving-cert" not found
Jul 15 21:16:38.005652 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:38.005622    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bcdc6d6-3433-424b-b134-9a63e3df4bb4-openshift-service-ca\") pod \"maas-api-e2e-migrate-baea565a-7764cdfdcb-26tfv\" (UID: \"2bcdc6d6-3433-424b-b134-9a63e3df4bb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-baea565a-7764cdfdcb-26tfv"
Jul 15 21:16:38.018810 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:38.018784    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp7kl\" (UniqueName: \"kubernetes.io/projected/2bcdc6d6-3433-424b-b134-9a63e3df4bb4-kube-api-access-vp7kl\") pod \"maas-api-e2e-migrate-baea565a-7764cdfdcb-26tfv\" (UID: \"2bcdc6d6-3433-424b-b134-9a63e3df4bb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-baea565a-7764cdfdcb-26tfv"
Jul 15 21:16:38.068046 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:38.068014    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:16:38.222161 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:38.222121    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7"
Jul 15 21:16:38.223140 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:38.223121    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7"
Jul 15 21:16:38.391053 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:38.391015    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:16:38.491067 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:38.491031    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:16:38.510291 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:38.510263    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bcdc6d6-3433-424b-b134-9a63e3df4bb4-maas-api-tls\") pod \"maas-api-e2e-migrate-baea565a-7764cdfdcb-26tfv\" (UID: \"2bcdc6d6-3433-424b-b134-9a63e3df4bb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-baea565a-7764cdfdcb-26tfv"
Jul 15 21:16:38.512706 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:38.512677    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bcdc6d6-3433-424b-b134-9a63e3df4bb4-maas-api-tls\") pod \"maas-api-e2e-migrate-baea565a-7764cdfdcb-26tfv\" (UID: \"2bcdc6d6-3433-424b-b134-9a63e3df4bb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-baea565a-7764cdfdcb-26tfv"
Jul 15 21:16:38.538585 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:38.538507    2566 patch_prober.go:28] interesting pod/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 15 21:16:38.538585 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:38.538570    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt" podUID="0f7d9980-d5ed-4368-9f06-4eb0885b2e9d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 15 21:16:38.772827 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:38.772779    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-baea565a-7764cdfdcb-26tfv"
Jul 15 21:16:38.908779 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:38.908753    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-baea565a-7764cdfdcb-26tfv"]
Jul 15 21:16:38.910914 ip-10-0-137-234 kubenswrapper[2566]: W0715 21:16:38.910859    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bcdc6d6_3433_424b_b134_9a63e3df4bb4.slice/crio-d01b41886ded544a08b304941d6d5a26521842170f6918dbcea541da17d8afe1 WatchSource:0}: Error finding container d01b41886ded544a08b304941d6d5a26521842170f6918dbcea541da17d8afe1: Status 404 returned error can't find the container with id d01b41886ded544a08b304941d6d5a26521842170f6918dbcea541da17d8afe1
Jul 15 21:16:39.227943 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:39.227903    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-baea565a-7764cdfdcb-26tfv" event={"ID":"2bcdc6d6-3433-424b-b134-9a63e3df4bb4","Type":"ContainerStarted","Data":"d01b41886ded544a08b304941d6d5a26521842170f6918dbcea541da17d8afe1"}
Jul 15 21:16:40.233784 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:40.233739    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-baea565a-7764cdfdcb-26tfv" event={"ID":"2bcdc6d6-3433-424b-b134-9a63e3df4bb4","Type":"ContainerStarted","Data":"2b1150360078eb3e93a643f3a97714aa0539202b263327e661d23f76699af362"}
Jul 15 21:16:40.234247 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:40.233922    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-baea565a-7764cdfdcb-26tfv"
Jul 15 21:16:40.253752 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:40.253699    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-baea565a-7764cdfdcb-26tfv" podStartSLOduration=2.828427787 podStartE2EDuration="3.253684156s" totalImagesPullingTime="425.256369ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:16:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:16:40.252495323 +0000 UTC m=+1733.401533379" watchObservedRunningTime="2026-07-15 21:16:40.253684156 +0000 UTC m=+1733.402722213"
Jul 15 21:16:41.072004 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:41.071939    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt"
Jul 15 21:16:41.136054 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:41.136030    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-credential-socket\") pod \"0f7d9980-d5ed-4368-9f06-4eb0885b2e9d\" (UID: \"0f7d9980-d5ed-4368-9f06-4eb0885b2e9d\") "
Jul 15 21:16:41.136150 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:41.136066    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzhbw\" (UniqueName: \"kubernetes.io/projected/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-kube-api-access-nzhbw\") pod \"0f7d9980-d5ed-4368-9f06-4eb0885b2e9d\" (UID: \"0f7d9980-d5ed-4368-9f06-4eb0885b2e9d\") "
Jul 15 21:16:41.136150 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:41.136086    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-workload-certs\") pod \"0f7d9980-d5ed-4368-9f06-4eb0885b2e9d\" (UID: \"0f7d9980-d5ed-4368-9f06-4eb0885b2e9d\") "
Jul 15 21:16:41.136150 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:41.136111    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-istio-data\") pod \"0f7d9980-d5ed-4368-9f06-4eb0885b2e9d\" (UID: \"0f7d9980-d5ed-4368-9f06-4eb0885b2e9d\") "
Jul 15 21:16:41.136300 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:41.136169    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-workload-socket\") pod \"0f7d9980-d5ed-4368-9f06-4eb0885b2e9d\" (UID: \"0f7d9980-d5ed-4368-9f06-4eb0885b2e9d\") "
Jul 15 21:16:41.136300 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:41.136201    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-istio-podinfo\") pod \"0f7d9980-d5ed-4368-9f06-4eb0885b2e9d\" (UID: \"0f7d9980-d5ed-4368-9f06-4eb0885b2e9d\") "
Jul 15 21:16:41.136300 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:41.136259    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-istio-envoy\") pod \"0f7d9980-d5ed-4368-9f06-4eb0885b2e9d\" (UID: \"0f7d9980-d5ed-4368-9f06-4eb0885b2e9d\") "
Jul 15 21:16:41.136300 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:41.136290    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-istiod-ca-cert\") pod \"0f7d9980-d5ed-4368-9f06-4eb0885b2e9d\" (UID: \"0f7d9980-d5ed-4368-9f06-4eb0885b2e9d\") "
Jul 15 21:16:41.136500 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:41.136336    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-istio-token\") pod \"0f7d9980-d5ed-4368-9f06-4eb0885b2e9d\" (UID: \"0f7d9980-d5ed-4368-9f06-4eb0885b2e9d\") "
Jul 15 21:16:41.136500 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:41.136336    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0f7d9980-d5ed-4368-9f06-4eb0885b2e9d" (UID: "0f7d9980-d5ed-4368-9f06-4eb0885b2e9d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:16:41.136715 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:41.136696    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-workload-certs\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:16:41.136773 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:41.136349    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0f7d9980-d5ed-4368-9f06-4eb0885b2e9d" (UID: "0f7d9980-d5ed-4368-9f06-4eb0885b2e9d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:16:41.136829 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:41.136775    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0f7d9980-d5ed-4368-9f06-4eb0885b2e9d" (UID: "0f7d9980-d5ed-4368-9f06-4eb0885b2e9d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:16:41.137232 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:41.137013    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0f7d9980-d5ed-4368-9f06-4eb0885b2e9d" (UID: "0f7d9980-d5ed-4368-9f06-4eb0885b2e9d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 21:16:41.137232 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:41.137210    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-istio-data" (OuterVolumeSpecName: "istio-data") pod "0f7d9980-d5ed-4368-9f06-4eb0885b2e9d" (UID: "0f7d9980-d5ed-4368-9f06-4eb0885b2e9d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:16:41.138726 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:41.138658    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0f7d9980-d5ed-4368-9f06-4eb0885b2e9d" (UID: "0f7d9980-d5ed-4368-9f06-4eb0885b2e9d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:16:41.138830 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:41.138798    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-kube-api-access-nzhbw" (OuterVolumeSpecName: "kube-api-access-nzhbw") pod "0f7d9980-d5ed-4368-9f06-4eb0885b2e9d" (UID: "0f7d9980-d5ed-4368-9f06-4eb0885b2e9d"). InnerVolumeSpecName "kube-api-access-nzhbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 21:16:41.139005 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:41.138827    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-istio-token" (OuterVolumeSpecName: "istio-token") pod "0f7d9980-d5ed-4368-9f06-4eb0885b2e9d" (UID: "0f7d9980-d5ed-4368-9f06-4eb0885b2e9d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 21:16:41.139229 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:41.139199    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0f7d9980-d5ed-4368-9f06-4eb0885b2e9d" (UID: "0f7d9980-d5ed-4368-9f06-4eb0885b2e9d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 15 21:16:41.237952 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:41.237922    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-istio-token\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:16:41.237952 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:41.237944    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-credential-socket\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:16:41.237952 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:41.237953    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzhbw\" (UniqueName: \"kubernetes.io/projected/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-kube-api-access-nzhbw\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:16:41.238380 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:41.237963    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-istio-data\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:16:41.238380 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:41.237973    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-workload-socket\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:16:41.238380 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:41.237981    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-istio-podinfo\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:16:41.238380 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:41.237988    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-istio-envoy\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:16:41.238380 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:41.237996    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d-istiod-ca-cert\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:16:41.240565 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:41.240539    2566 generic.go:356] "Generic (PLEG): container finished" podID="0f7d9980-d5ed-4368-9f06-4eb0885b2e9d" containerID="92a94f400f8b69de3ccc0c300d6ff54b729f7e4e0b2d09403e54d99ef5ea3aee" exitCode=0
Jul 15 21:16:41.240678 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:41.240590    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt" event={"ID":"0f7d9980-d5ed-4368-9f06-4eb0885b2e9d","Type":"ContainerDied","Data":"92a94f400f8b69de3ccc0c300d6ff54b729f7e4e0b2d09403e54d99ef5ea3aee"}
Jul 15 21:16:41.240678 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:41.240611    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt"
Jul 15 21:16:41.240678 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:41.240646    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt" event={"ID":"0f7d9980-d5ed-4368-9f06-4eb0885b2e9d","Type":"ContainerDied","Data":"f47b50e114df46286b4bae10faa5738d74c6eafe33d04608e1a559292b0a6f08"}
Jul 15 21:16:41.240678 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:41.240667    2566 scope.go:117] "RemoveContainer" containerID="92a94f400f8b69de3ccc0c300d6ff54b729f7e4e0b2d09403e54d99ef5ea3aee"
Jul 15 21:16:41.249839 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:41.249821    2566 scope.go:117] "RemoveContainer" containerID="aaaf95c9643d2817557fd54b18b642498c0928141d37a82d2f6c638bafc351bb"
Jul 15 21:16:41.257998 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:41.257982    2566 scope.go:117] "RemoveContainer" containerID="92a94f400f8b69de3ccc0c300d6ff54b729f7e4e0b2d09403e54d99ef5ea3aee"
Jul 15 21:16:41.258250 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:16:41.258227    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92a94f400f8b69de3ccc0c300d6ff54b729f7e4e0b2d09403e54d99ef5ea3aee\": container with ID starting with 92a94f400f8b69de3ccc0c300d6ff54b729f7e4e0b2d09403e54d99ef5ea3aee not found: ID does not exist" containerID="92a94f400f8b69de3ccc0c300d6ff54b729f7e4e0b2d09403e54d99ef5ea3aee"
Jul 15 21:16:41.258301 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:41.258258    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92a94f400f8b69de3ccc0c300d6ff54b729f7e4e0b2d09403e54d99ef5ea3aee"} err="failed to get container status \"92a94f400f8b69de3ccc0c300d6ff54b729f7e4e0b2d09403e54d99ef5ea3aee\": rpc error: code = NotFound desc = could not find container \"92a94f400f8b69de3ccc0c300d6ff54b729f7e4e0b2d09403e54d99ef5ea3aee\": container with ID starting with 92a94f400f8b69de3ccc0c300d6ff54b729f7e4e0b2d09403e54d99ef5ea3aee not found: ID does not exist"
Jul 15 21:16:41.258301 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:41.258276    2566 scope.go:117] "RemoveContainer" containerID="aaaf95c9643d2817557fd54b18b642498c0928141d37a82d2f6c638bafc351bb"
Jul 15 21:16:41.258548 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:16:41.258528    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaaf95c9643d2817557fd54b18b642498c0928141d37a82d2f6c638bafc351bb\": container with ID starting with aaaf95c9643d2817557fd54b18b642498c0928141d37a82d2f6c638bafc351bb not found: ID does not exist" containerID="aaaf95c9643d2817557fd54b18b642498c0928141d37a82d2f6c638bafc351bb"
Jul 15 21:16:41.258598 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:41.258553    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaaf95c9643d2817557fd54b18b642498c0928141d37a82d2f6c638bafc351bb"} err="failed to get container status \"aaaf95c9643d2817557fd54b18b642498c0928141d37a82d2f6c638bafc351bb\": rpc error: code = NotFound desc = could not find container \"aaaf95c9643d2817557fd54b18b642498c0928141d37a82d2f6c638bafc351bb\": container with ID starting with aaaf95c9643d2817557fd54b18b642498c0928141d37a82d2f6c638bafc351bb not found: ID does not exist"
Jul 15 21:16:41.263714 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:41.263693    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt"]
Jul 15 21:16:41.267603 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:41.267585    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-8143c542-openshift-default-855dffdb8c-rl5zt"]
Jul 15 21:16:41.756867 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:41.756837    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f7d9980-d5ed-4368-9f06-4eb0885b2e9d" path="/var/lib/kubelet/pods/0f7d9980-d5ed-4368-9f06-4eb0885b2e9d/volumes"
Jul 15 21:16:42.236660 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:42.236626    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-baea565a-zgr74"]
Jul 15 21:16:42.237060 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:42.237042    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f7d9980-d5ed-4368-9f06-4eb0885b2e9d" containerName="istio-proxy"
Jul 15 21:16:42.237114 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:42.237063    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f7d9980-d5ed-4368-9f06-4eb0885b2e9d" containerName="istio-proxy"
Jul 15 21:16:42.237146 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:42.237137    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f7d9980-d5ed-4368-9f06-4eb0885b2e9d" containerName="istio-proxy"
Jul 15 21:16:42.237179 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:42.237146    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f7d9980-d5ed-4368-9f06-4eb0885b2e9d" containerName="istio-proxy"
Jul 15 21:16:42.241590 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:42.241571    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-baea565a-zgr74"
Jul 15 21:16:42.244475 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:42.244451    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zsrf5\""
Jul 15 21:16:42.249862 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:42.249839    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-baea565a-zgr74"]
Jul 15 21:16:42.348701 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:42.348663    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15409c13-7939-449c-8115-31e8fc4bddf1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-baea565a-zgr74\" (UID: \"15409c13-7939-449c-8115-31e8fc4bddf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-baea565a-zgr74"
Jul 15 21:16:42.449672 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:42.449639    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15409c13-7939-449c-8115-31e8fc4bddf1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-baea565a-zgr74\" (UID: \"15409c13-7939-449c-8115-31e8fc4bddf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-baea565a-zgr74"
Jul 15 21:16:42.450270 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:42.450248    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15409c13-7939-449c-8115-31e8fc4bddf1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-baea565a-zgr74\" (UID: \"15409c13-7939-449c-8115-31e8fc4bddf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-baea565a-zgr74"
Jul 15 21:16:42.553949 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:42.553846    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-baea565a-zgr74"
Jul 15 21:16:42.690861 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:42.690736    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-baea565a-zgr74"]
Jul 15 21:16:42.693580 ip-10-0-137-234 kubenswrapper[2566]: W0715 21:16:42.693549    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15409c13_7939_449c_8115_31e8fc4bddf1.slice/crio-f861f49b9e4be0c9e04feb9395fea4843a4f02cd412891986daf174327b117d6 WatchSource:0}: Error finding container f861f49b9e4be0c9e04feb9395fea4843a4f02cd412891986daf174327b117d6: Status 404 returned error can't find the container with id f861f49b9e4be0c9e04feb9395fea4843a4f02cd412891986daf174327b117d6
Jul 15 21:16:43.253351 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:43.253315    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-baea565a-zgr74" event={"ID":"15409c13-7939-449c-8115-31e8fc4bddf1","Type":"ContainerStarted","Data":"11537732f6068134983fd36736aeb96196158e612580ead20199b0a1d23d8921"}
Jul 15 21:16:43.253739 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:43.253412    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-baea565a-zgr74" event={"ID":"15409c13-7939-449c-8115-31e8fc4bddf1","Type":"ContainerStarted","Data":"f861f49b9e4be0c9e04feb9395fea4843a4f02cd412891986daf174327b117d6"}
Jul 15 21:16:43.271330 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:43.271281    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-baea565a-zgr74" podStartSLOduration=1.271265646 podStartE2EDuration="1.271265646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:16:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:16:43.269472138 +0000 UTC m=+1736.418510195" watchObservedRunningTime="2026-07-15 21:16:43.271265646 +0000 UTC m=+1736.420303702"
Jul 15 21:16:44.259647 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:44.259611    2566 generic.go:356] "Generic (PLEG): container finished" podID="15409c13-7939-449c-8115-31e8fc4bddf1" containerID="11537732f6068134983fd36736aeb96196158e612580ead20199b0a1d23d8921" exitCode=7
Jul 15 21:16:44.260028 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:44.259663    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-baea565a-zgr74" event={"ID":"15409c13-7939-449c-8115-31e8fc4bddf1","Type":"ContainerDied","Data":"11537732f6068134983fd36736aeb96196158e612580ead20199b0a1d23d8921"}
Jul 15 21:16:44.260028 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:44.259854    2566 scope.go:117] "RemoveContainer" containerID="11537732f6068134983fd36736aeb96196158e612580ead20199b0a1d23d8921"
Jul 15 21:16:45.266454 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:45.266365    2566 generic.go:356] "Generic (PLEG): container finished" podID="15409c13-7939-449c-8115-31e8fc4bddf1" containerID="b9c5c1130c58ad82c6fd87fa7acab0ddfd4f4a45001fd4953d7873138e7760ee" exitCode=7
Jul 15 21:16:45.266872 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:45.266448    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-baea565a-zgr74" event={"ID":"15409c13-7939-449c-8115-31e8fc4bddf1","Type":"ContainerDied","Data":"b9c5c1130c58ad82c6fd87fa7acab0ddfd4f4a45001fd4953d7873138e7760ee"}
Jul 15 21:16:45.266872 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:45.266510    2566 scope.go:117] "RemoveContainer" containerID="11537732f6068134983fd36736aeb96196158e612580ead20199b0a1d23d8921"
Jul 15 21:16:45.266872 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:45.266717    2566 scope.go:117] "RemoveContainer" containerID="b9c5c1130c58ad82c6fd87fa7acab0ddfd4f4a45001fd4953d7873138e7760ee"
Jul 15 21:16:45.267018 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:16:45.266932    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-baea565a-zgr74_odh-ai-gateway-infra(15409c13-7939-449c-8115-31e8fc4bddf1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-baea565a-zgr74" podUID="15409c13-7939-449c-8115-31e8fc4bddf1"
Jul 15 21:16:46.247665 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:46.247636    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-baea565a-7764cdfdcb-26tfv"
Jul 15 21:16:46.274096 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:46.274070    2566 scope.go:117] "RemoveContainer" containerID="b9c5c1130c58ad82c6fd87fa7acab0ddfd4f4a45001fd4953d7873138e7760ee"
Jul 15 21:16:46.274496 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:16:46.274259    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-baea565a-zgr74_odh-ai-gateway-infra(15409c13-7939-449c-8115-31e8fc4bddf1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-baea565a-zgr74" podUID="15409c13-7939-449c-8115-31e8fc4bddf1"
Jul 15 21:16:52.861270 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:52.861233    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:16:52.932574 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:52.932517    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:16:53.400692 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:53.400661    2566 patch_prober.go:28] interesting pod/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": read tcp 10.134.0.2:49566->10.134.0.37:15021: read: connection reset by peer" start-of-body=
Jul 15 21:16:53.400911 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:53.400722    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7" podUID="dc0cedd8-e570-4a06-b010-a1603c763de3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": read tcp 10.134.0.2:49566->10.134.0.37:15021: read: connection reset by peer"
Jul 15 21:16:54.322429 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:54.322394    2566 generic.go:356] "Generic (PLEG): container finished" podID="dc0cedd8-e570-4a06-b010-a1603c763de3" containerID="afc8cb0d7755dc3f9992a07144b576ddd231f498cb59ad2a2cf4cb97a9e24e39" exitCode=0
Jul 15 21:16:54.322792 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:54.322468    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7" event={"ID":"dc0cedd8-e570-4a06-b010-a1603c763de3","Type":"ContainerDied","Data":"afc8cb0d7755dc3f9992a07144b576ddd231f498cb59ad2a2cf4cb97a9e24e39"}
Jul 15 21:16:54.322885 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:54.322871    2566 scope.go:117] "RemoveContainer" containerID="afc8cb0d7755dc3f9992a07144b576ddd231f498cb59ad2a2cf4cb97a9e24e39"
Jul 15 21:16:54.323978 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:54.323952    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:16:54.324053 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:54.324036    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:16:54.324101 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:54.324085    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:16:55.328801 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:55.328767    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7" event={"ID":"dc0cedd8-e570-4a06-b010-a1603c763de3","Type":"ContainerStarted","Data":"da12250506b0cd41d7db6850ebaeb508bf8788cdaac3ebd9d2e2a775e42c51ee"}
Jul 15 21:16:55.727511 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:55.727467    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7"
Jul 15 21:16:55.727732 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:55.727707    2566 patch_prober.go:28] interesting pod/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Jul 15 21:16:55.727793 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:55.727773    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7" podUID="dc0cedd8-e570-4a06-b010-a1603c763de3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Jul 15 21:16:56.728180 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:56.728139    2566 patch_prober.go:28] interesting pod/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Jul 15 21:16:56.728661 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:56.728217    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7" podUID="dc0cedd8-e570-4a06-b010-a1603c763de3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Jul 15 21:16:57.136741 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:57.136689    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:16:57.728559 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:57.728525    2566 patch_prober.go:28] interesting pod/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Jul 15 21:16:57.728997 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:57.728583    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7" podUID="dc0cedd8-e570-4a06-b010-a1603c763de3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Jul 15 21:16:57.753369 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:57.753338    2566 scope.go:117] "RemoveContainer" containerID="b9c5c1130c58ad82c6fd87fa7acab0ddfd4f4a45001fd4953d7873138e7760ee"
Jul 15 21:16:58.344384 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:58.344296    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-baea565a-zgr74" event={"ID":"15409c13-7939-449c-8115-31e8fc4bddf1","Type":"ContainerStarted","Data":"4f454abb69a30d1c9f3d062751529a4136b915a93521eeb9d0832229b8998ca3"}
Jul 15 21:16:58.727571 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:58.727528    2566 patch_prober.go:28] interesting pod/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Jul 15 21:16:58.727754 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:58.727593    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7" podUID="dc0cedd8-e570-4a06-b010-a1603c763de3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Jul 15 21:16:59.351968 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:59.351932    2566 generic.go:356] "Generic (PLEG): container finished" podID="15409c13-7939-449c-8115-31e8fc4bddf1" containerID="4f454abb69a30d1c9f3d062751529a4136b915a93521eeb9d0832229b8998ca3" exitCode=0
Jul 15 21:16:59.352358 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:59.351974    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-baea565a-zgr74" event={"ID":"15409c13-7939-449c-8115-31e8fc4bddf1","Type":"ContainerDied","Data":"4f454abb69a30d1c9f3d062751529a4136b915a93521eeb9d0832229b8998ca3"}
Jul 15 21:16:59.352358 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:59.352046    2566 scope.go:117] "RemoveContainer" containerID="b9c5c1130c58ad82c6fd87fa7acab0ddfd4f4a45001fd4953d7873138e7760ee"
Jul 15 21:16:59.380194 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:59.380162    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-baea565a-zgr74"]
Jul 15 21:16:59.731578 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:59.731551    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7"
Jul 15 21:16:59.731947 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:59.731927    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7"
Jul 15 21:16:59.732679 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:16:59.732665    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7"
Jul 15 21:17:00.493723 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:00.493691    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-baea565a-zgr74"
Jul 15 21:17:00.627033 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:00.626947    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15409c13-7939-449c-8115-31e8fc4bddf1-maas-api-service-ca\") pod \"15409c13-7939-449c-8115-31e8fc4bddf1\" (UID: \"15409c13-7939-449c-8115-31e8fc4bddf1\") "
Jul 15 21:17:00.627338 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:00.627317    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15409c13-7939-449c-8115-31e8fc4bddf1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "15409c13-7939-449c-8115-31e8fc4bddf1" (UID: "15409c13-7939-449c-8115-31e8fc4bddf1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 21:17:00.728096 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:00.728060    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15409c13-7939-449c-8115-31e8fc4bddf1-maas-api-service-ca\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:17:01.364586 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:01.364556    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-baea565a-zgr74"
Jul 15 21:17:01.364767 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:01.364603    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-baea565a-zgr74" event={"ID":"15409c13-7939-449c-8115-31e8fc4bddf1","Type":"ContainerDied","Data":"f861f49b9e4be0c9e04feb9395fea4843a4f02cd412891986daf174327b117d6"}
Jul 15 21:17:01.364767 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:01.364661    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f861f49b9e4be0c9e04feb9395fea4843a4f02cd412891986daf174327b117d6"
Jul 15 21:17:01.396125 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:01.396092    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-baea565a-zgr74"]
Jul 15 21:17:01.404116 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:01.404091    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-baea565a-zgr74"]
Jul 15 21:17:01.756497 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:01.756452    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15409c13-7939-449c-8115-31e8fc4bddf1" path="/var/lib/kubelet/pods/15409c13-7939-449c-8115-31e8fc4bddf1/volumes"
Jul 15 21:17:32.294345 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:32.294309    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-baea565a-jbcqp"]
Jul 15 21:17:32.294812 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:32.294665    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15409c13-7939-449c-8115-31e8fc4bddf1" containerName="revoke-keys"
Jul 15 21:17:32.294812 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:32.294676    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="15409c13-7939-449c-8115-31e8fc4bddf1" containerName="revoke-keys"
Jul 15 21:17:32.294812 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:32.294687    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15409c13-7939-449c-8115-31e8fc4bddf1" containerName="revoke-keys"
Jul 15 21:17:32.294812 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:32.294692    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="15409c13-7939-449c-8115-31e8fc4bddf1" containerName="revoke-keys"
Jul 15 21:17:32.294812 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:32.294713    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f7d9980-d5ed-4368-9f06-4eb0885b2e9d" containerName="istio-proxy"
Jul 15 21:17:32.294812 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:32.294718    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f7d9980-d5ed-4368-9f06-4eb0885b2e9d" containerName="istio-proxy"
Jul 15 21:17:32.294812 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:32.294767    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="15409c13-7939-449c-8115-31e8fc4bddf1" containerName="revoke-keys"
Jul 15 21:17:32.294812 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:32.294778    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="15409c13-7939-449c-8115-31e8fc4bddf1" containerName="revoke-keys"
Jul 15 21:17:32.294812 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:32.294783    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="15409c13-7939-449c-8115-31e8fc4bddf1" containerName="revoke-keys"
Jul 15 21:17:32.296697 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:32.296679    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-baea565a-jbcqp"
Jul 15 21:17:32.300027 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:32.300004    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zsrf5\""
Jul 15 21:17:32.308324 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:32.308294    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-baea565a-jbcqp"]
Jul 15 21:17:32.403517 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:32.403478    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab0b03e8-299b-4e2a-a227-ba1d011fac09-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-baea565a-jbcqp\" (UID: \"ab0b03e8-299b-4e2a-a227-ba1d011fac09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-baea565a-jbcqp"
Jul 15 21:17:32.504698 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:32.504660    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab0b03e8-299b-4e2a-a227-ba1d011fac09-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-baea565a-jbcqp\" (UID: \"ab0b03e8-299b-4e2a-a227-ba1d011fac09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-baea565a-jbcqp"
Jul 15 21:17:32.505297 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:32.505278    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab0b03e8-299b-4e2a-a227-ba1d011fac09-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-baea565a-jbcqp\" (UID: \"ab0b03e8-299b-4e2a-a227-ba1d011fac09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-baea565a-jbcqp"
Jul 15 21:17:32.607220 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:32.607129    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-baea565a-jbcqp"
Jul 15 21:17:32.747332 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:32.747306    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-baea565a-jbcqp"]
Jul 15 21:17:32.749367 ip-10-0-137-234 kubenswrapper[2566]: W0715 21:17:32.749337    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab0b03e8_299b_4e2a_a227_ba1d011fac09.slice/crio-18a5b66b29c716b1f2d4df536a2602208a536062aab2e4c82aa8cdc9baf4ee1b WatchSource:0}: Error finding container 18a5b66b29c716b1f2d4df536a2602208a536062aab2e4c82aa8cdc9baf4ee1b: Status 404 returned error can't find the container with id 18a5b66b29c716b1f2d4df536a2602208a536062aab2e4c82aa8cdc9baf4ee1b
Jul 15 21:17:33.511434 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:33.511403    2566 generic.go:356] "Generic (PLEG): container finished" podID="ab0b03e8-299b-4e2a-a227-ba1d011fac09" containerID="0a7cefaa1d8553e9318478399863fd63ca381a570dbe303093ca7c789bed5cb6" exitCode=0
Jul 15 21:17:33.511803 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:33.511449    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-baea565a-jbcqp" event={"ID":"ab0b03e8-299b-4e2a-a227-ba1d011fac09","Type":"ContainerDied","Data":"0a7cefaa1d8553e9318478399863fd63ca381a570dbe303093ca7c789bed5cb6"}
Jul 15 21:17:33.511803 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:33.511490    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-baea565a-jbcqp" event={"ID":"ab0b03e8-299b-4e2a-a227-ba1d011fac09","Type":"ContainerStarted","Data":"18a5b66b29c716b1f2d4df536a2602208a536062aab2e4c82aa8cdc9baf4ee1b"}
Jul 15 21:17:34.648619 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:34.648595    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-baea565a-jbcqp"
Jul 15 21:17:34.726084 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:34.726053    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab0b03e8-299b-4e2a-a227-ba1d011fac09-maas-api-service-ca\") pod \"ab0b03e8-299b-4e2a-a227-ba1d011fac09\" (UID: \"ab0b03e8-299b-4e2a-a227-ba1d011fac09\") "
Jul 15 21:17:34.726405 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:34.726374    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab0b03e8-299b-4e2a-a227-ba1d011fac09-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ab0b03e8-299b-4e2a-a227-ba1d011fac09" (UID: "ab0b03e8-299b-4e2a-a227-ba1d011fac09"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 21:17:34.826722 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:34.826618    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab0b03e8-299b-4e2a-a227-ba1d011fac09-maas-api-service-ca\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:17:35.522702 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:35.522670    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-baea565a-jbcqp"
Jul 15 21:17:35.522930 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:35.522667    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-baea565a-jbcqp" event={"ID":"ab0b03e8-299b-4e2a-a227-ba1d011fac09","Type":"ContainerDied","Data":"18a5b66b29c716b1f2d4df536a2602208a536062aab2e4c82aa8cdc9baf4ee1b"}
Jul 15 21:17:35.522930 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:35.522798    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18a5b66b29c716b1f2d4df536a2602208a536062aab2e4c82aa8cdc9baf4ee1b"
Jul 15 21:17:42.367361 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:42.367273    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7"]
Jul 15 21:17:42.367760 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:42.367507    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7" podUID="dc0cedd8-e570-4a06-b010-a1603c763de3" containerName="istio-proxy" containerID="cri-o://da12250506b0cd41d7db6850ebaeb508bf8788cdaac3ebd9d2e2a775e42c51ee" gracePeriod=30
Jul 15 21:17:42.408628 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:42.408580    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:17:42.513774 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:42.513739    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:17:42.910482 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:42.910448    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-baea565a-7764cdfdcb-26tfv"]
Jul 15 21:17:42.910721 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:42.910694    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-baea565a-7764cdfdcb-26tfv" podUID="2bcdc6d6-3433-424b-b134-9a63e3df4bb4" containerName="maas-api" containerID="cri-o://2b1150360078eb3e93a643f3a97714aa0539202b263327e661d23f76699af362" gracePeriod=30
Jul 15 21:17:42.938153 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:42.938111    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:17:43.162470 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:43.162403    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-baea565a-7764cdfdcb-26tfv"
Jul 15 21:17:43.299451 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:43.299417    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bcdc6d6-3433-424b-b134-9a63e3df4bb4-openshift-service-ca\") pod \"2bcdc6d6-3433-424b-b134-9a63e3df4bb4\" (UID: \"2bcdc6d6-3433-424b-b134-9a63e3df4bb4\") "
Jul 15 21:17:43.299611 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:43.299483    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bcdc6d6-3433-424b-b134-9a63e3df4bb4-maas-api-tls\") pod \"2bcdc6d6-3433-424b-b134-9a63e3df4bb4\" (UID: \"2bcdc6d6-3433-424b-b134-9a63e3df4bb4\") "
Jul 15 21:17:43.299611 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:43.299523    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vp7kl\" (UniqueName: \"kubernetes.io/projected/2bcdc6d6-3433-424b-b134-9a63e3df4bb4-kube-api-access-vp7kl\") pod \"2bcdc6d6-3433-424b-b134-9a63e3df4bb4\" (UID: \"2bcdc6d6-3433-424b-b134-9a63e3df4bb4\") "
Jul 15 21:17:43.299842 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:43.299813    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bcdc6d6-3433-424b-b134-9a63e3df4bb4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2bcdc6d6-3433-424b-b134-9a63e3df4bb4" (UID: "2bcdc6d6-3433-424b-b134-9a63e3df4bb4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 21:17:43.301586 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:43.301558    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bcdc6d6-3433-424b-b134-9a63e3df4bb4-kube-api-access-vp7kl" (OuterVolumeSpecName: "kube-api-access-vp7kl") pod "2bcdc6d6-3433-424b-b134-9a63e3df4bb4" (UID: "2bcdc6d6-3433-424b-b134-9a63e3df4bb4"). InnerVolumeSpecName "kube-api-access-vp7kl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 21:17:43.301714 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:43.301686    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bcdc6d6-3433-424b-b134-9a63e3df4bb4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2bcdc6d6-3433-424b-b134-9a63e3df4bb4" (UID: "2bcdc6d6-3433-424b-b134-9a63e3df4bb4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 21:17:43.400341 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:43.400302    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bcdc6d6-3433-424b-b134-9a63e3df4bb4-maas-api-tls\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:17:43.400341 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:43.400333    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp7kl\" (UniqueName: \"kubernetes.io/projected/2bcdc6d6-3433-424b-b134-9a63e3df4bb4-kube-api-access-vp7kl\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:17:43.400341 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:43.400349    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bcdc6d6-3433-424b-b134-9a63e3df4bb4-openshift-service-ca\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:17:43.562754 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:43.562719    2566 generic.go:356] "Generic (PLEG): container finished" podID="2bcdc6d6-3433-424b-b134-9a63e3df4bb4" containerID="2b1150360078eb3e93a643f3a97714aa0539202b263327e661d23f76699af362" exitCode=0
Jul 15 21:17:43.562930 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:43.562764    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-baea565a-7764cdfdcb-26tfv"
Jul 15 21:17:43.562930 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:43.562807    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-baea565a-7764cdfdcb-26tfv" event={"ID":"2bcdc6d6-3433-424b-b134-9a63e3df4bb4","Type":"ContainerDied","Data":"2b1150360078eb3e93a643f3a97714aa0539202b263327e661d23f76699af362"}
Jul 15 21:17:43.562930 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:43.562860    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-baea565a-7764cdfdcb-26tfv" event={"ID":"2bcdc6d6-3433-424b-b134-9a63e3df4bb4","Type":"ContainerDied","Data":"d01b41886ded544a08b304941d6d5a26521842170f6918dbcea541da17d8afe1"}
Jul 15 21:17:43.562930 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:43.562890    2566 scope.go:117] "RemoveContainer" containerID="2b1150360078eb3e93a643f3a97714aa0539202b263327e661d23f76699af362"
Jul 15 21:17:43.572285 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:43.572267    2566 scope.go:117] "RemoveContainer" containerID="2b1150360078eb3e93a643f3a97714aa0539202b263327e661d23f76699af362"
Jul 15 21:17:43.572563 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:17:43.572543    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b1150360078eb3e93a643f3a97714aa0539202b263327e661d23f76699af362\": container with ID starting with 2b1150360078eb3e93a643f3a97714aa0539202b263327e661d23f76699af362 not found: ID does not exist" containerID="2b1150360078eb3e93a643f3a97714aa0539202b263327e661d23f76699af362"
Jul 15 21:17:43.572631 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:43.572576    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b1150360078eb3e93a643f3a97714aa0539202b263327e661d23f76699af362"} err="failed to get container status \"2b1150360078eb3e93a643f3a97714aa0539202b263327e661d23f76699af362\": rpc error: code = NotFound desc = could not find container \"2b1150360078eb3e93a643f3a97714aa0539202b263327e661d23f76699af362\": container with ID starting with 2b1150360078eb3e93a643f3a97714aa0539202b263327e661d23f76699af362 not found: ID does not exist"
Jul 15 21:17:43.585509 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:43.585486    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-baea565a-7764cdfdcb-26tfv"]
Jul 15 21:17:43.589072 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:43.589054    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-baea565a-7764cdfdcb-26tfv"]
Jul 15 21:17:43.757099 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:43.757064    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bcdc6d6-3433-424b-b134-9a63e3df4bb4" path="/var/lib/kubelet/pods/2bcdc6d6-3433-424b-b134-9a63e3df4bb4/volumes"
Jul 15 21:17:44.733976 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:44.733939    2566 patch_prober.go:28] interesting pod/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 15 21:17:44.734370 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:44.734005    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7" podUID="dc0cedd8-e570-4a06-b010-a1603c763de3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 15 21:17:47.589843 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:47.589802    2566 generic.go:356] "Generic (PLEG): container finished" podID="dc0cedd8-e570-4a06-b010-a1603c763de3" containerID="da12250506b0cd41d7db6850ebaeb508bf8788cdaac3ebd9d2e2a775e42c51ee" exitCode=0
Jul 15 21:17:47.590318 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:47.589890    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7" event={"ID":"dc0cedd8-e570-4a06-b010-a1603c763de3","Type":"ContainerDied","Data":"da12250506b0cd41d7db6850ebaeb508bf8788cdaac3ebd9d2e2a775e42c51ee"}
Jul 15 21:17:47.590318 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:47.590018    2566 scope.go:117] "RemoveContainer" containerID="afc8cb0d7755dc3f9992a07144b576ddd231f498cb59ad2a2cf4cb97a9e24e39"
Jul 15 21:17:47.652140 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:47.652114    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7"
Jul 15 21:17:47.738195 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:47.738161    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc0cedd8-e570-4a06-b010-a1603c763de3-istiod-ca-cert\") pod \"dc0cedd8-e570-4a06-b010-a1603c763de3\" (UID: \"dc0cedd8-e570-4a06-b010-a1603c763de3\") "
Jul 15 21:17:47.738195 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:47.738200    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc0cedd8-e570-4a06-b010-a1603c763de3-istio-data\") pod \"dc0cedd8-e570-4a06-b010-a1603c763de3\" (UID: \"dc0cedd8-e570-4a06-b010-a1603c763de3\") "
Jul 15 21:17:47.738428 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:47.738236    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc0cedd8-e570-4a06-b010-a1603c763de3-workload-certs\") pod \"dc0cedd8-e570-4a06-b010-a1603c763de3\" (UID: \"dc0cedd8-e570-4a06-b010-a1603c763de3\") "
Jul 15 21:17:47.738557 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:47.738511    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc0cedd8-e570-4a06-b010-a1603c763de3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dc0cedd8-e570-4a06-b010-a1603c763de3" (UID: "dc0cedd8-e570-4a06-b010-a1603c763de3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:17:47.738685 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:47.738595    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc0cedd8-e570-4a06-b010-a1603c763de3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dc0cedd8-e570-4a06-b010-a1603c763de3" (UID: "dc0cedd8-e570-4a06-b010-a1603c763de3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 21:17:47.738685 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:47.738655    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc0cedd8-e570-4a06-b010-a1603c763de3-istio-podinfo\") pod \"dc0cedd8-e570-4a06-b010-a1603c763de3\" (UID: \"dc0cedd8-e570-4a06-b010-a1603c763de3\") "
Jul 15 21:17:47.738809 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:47.738701    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc0cedd8-e570-4a06-b010-a1603c763de3-istio-token\") pod \"dc0cedd8-e570-4a06-b010-a1603c763de3\" (UID: \"dc0cedd8-e570-4a06-b010-a1603c763de3\") "
Jul 15 21:17:47.738809 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:47.738737    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc0cedd8-e570-4a06-b010-a1603c763de3-credential-socket\") pod \"dc0cedd8-e570-4a06-b010-a1603c763de3\" (UID: \"dc0cedd8-e570-4a06-b010-a1603c763de3\") "
Jul 15 21:17:47.738809 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:47.738768    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc0cedd8-e570-4a06-b010-a1603c763de3-workload-socket\") pod \"dc0cedd8-e570-4a06-b010-a1603c763de3\" (UID: \"dc0cedd8-e570-4a06-b010-a1603c763de3\") "
Jul 15 21:17:47.738809 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:47.738800    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vv8ft\" (UniqueName: \"kubernetes.io/projected/dc0cedd8-e570-4a06-b010-a1603c763de3-kube-api-access-vv8ft\") pod \"dc0cedd8-e570-4a06-b010-a1603c763de3\" (UID: \"dc0cedd8-e570-4a06-b010-a1603c763de3\") "
Jul 15 21:17:47.739041 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:47.738830    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc0cedd8-e570-4a06-b010-a1603c763de3-istio-envoy\") pod \"dc0cedd8-e570-4a06-b010-a1603c763de3\" (UID: \"dc0cedd8-e570-4a06-b010-a1603c763de3\") "
Jul 15 21:17:47.739041 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:47.739012    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc0cedd8-e570-4a06-b010-a1603c763de3-istio-data" (OuterVolumeSpecName: "istio-data") pod "dc0cedd8-e570-4a06-b010-a1603c763de3" (UID: "dc0cedd8-e570-4a06-b010-a1603c763de3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:17:47.739187 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:47.739164    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc0cedd8-e570-4a06-b010-a1603c763de3-istiod-ca-cert\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:17:47.739252 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:47.739192    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc0cedd8-e570-4a06-b010-a1603c763de3-istio-data\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:17:47.739252 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:47.739208    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc0cedd8-e570-4a06-b010-a1603c763de3-workload-certs\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:17:47.739421 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:47.739366    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc0cedd8-e570-4a06-b010-a1603c763de3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dc0cedd8-e570-4a06-b010-a1603c763de3" (UID: "dc0cedd8-e570-4a06-b010-a1603c763de3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:17:47.739421 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:47.739404    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc0cedd8-e570-4a06-b010-a1603c763de3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dc0cedd8-e570-4a06-b010-a1603c763de3" (UID: "dc0cedd8-e570-4a06-b010-a1603c763de3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:17:47.741241 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:47.741218    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc0cedd8-e570-4a06-b010-a1603c763de3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dc0cedd8-e570-4a06-b010-a1603c763de3" (UID: "dc0cedd8-e570-4a06-b010-a1603c763de3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:17:47.741671 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:47.741645    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc0cedd8-e570-4a06-b010-a1603c763de3-istio-token" (OuterVolumeSpecName: "istio-token") pod "dc0cedd8-e570-4a06-b010-a1603c763de3" (UID: "dc0cedd8-e570-4a06-b010-a1603c763de3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 21:17:47.741776 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:47.741714    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc0cedd8-e570-4a06-b010-a1603c763de3-kube-api-access-vv8ft" (OuterVolumeSpecName: "kube-api-access-vv8ft") pod "dc0cedd8-e570-4a06-b010-a1603c763de3" (UID: "dc0cedd8-e570-4a06-b010-a1603c763de3"). InnerVolumeSpecName "kube-api-access-vv8ft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 21:17:47.741951 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:47.741933    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc0cedd8-e570-4a06-b010-a1603c763de3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dc0cedd8-e570-4a06-b010-a1603c763de3" (UID: "dc0cedd8-e570-4a06-b010-a1603c763de3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 15 21:17:47.763668 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:47.763646    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-652lk_f9a159bb-39c4-4cca-92ff-1c74d739e311/ovn-acl-logging/0.log"
Jul 15 21:17:47.767809 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:47.767788    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-652lk_f9a159bb-39c4-4cca-92ff-1c74d739e311/ovn-acl-logging/0.log"
Jul 15 21:17:47.839656 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:47.839622    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc0cedd8-e570-4a06-b010-a1603c763de3-istio-podinfo\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:17:47.839656 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:47.839647    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc0cedd8-e570-4a06-b010-a1603c763de3-istio-token\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:17:47.839656 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:47.839658    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc0cedd8-e570-4a06-b010-a1603c763de3-credential-socket\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:17:47.839656 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:47.839666    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc0cedd8-e570-4a06-b010-a1603c763de3-workload-socket\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:17:47.840048 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:47.839675    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vv8ft\" (UniqueName: \"kubernetes.io/projected/dc0cedd8-e570-4a06-b010-a1603c763de3-kube-api-access-vv8ft\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:17:47.840048 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:47.839687    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc0cedd8-e570-4a06-b010-a1603c763de3-istio-envoy\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:17:48.596504 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:48.596476    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7"
Jul 15 21:17:48.597018 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:48.596478    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7" event={"ID":"dc0cedd8-e570-4a06-b010-a1603c763de3","Type":"ContainerDied","Data":"44ffa33c807d7e7a3d92509eaa087f246f8fbd9d637eff38e02901ddb95592ba"}
Jul 15 21:17:48.597018 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:48.596623    2566 scope.go:117] "RemoveContainer" containerID="da12250506b0cd41d7db6850ebaeb508bf8788cdaac3ebd9d2e2a775e42c51ee"
Jul 15 21:17:48.616976 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:48.616948    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7"]
Jul 15 21:17:48.618796 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:48.618771    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-baea565a-gw-openshift-default-797c7bd679-bnrg7"]
Jul 15 21:17:48.636853 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:48.636825    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:17:48.714739 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:48.714706    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:17:48.785628 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:48.785593    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk"]
Jul 15 21:17:48.786306 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:48.786281    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15409c13-7939-449c-8115-31e8fc4bddf1" containerName="revoke-keys"
Jul 15 21:17:48.786306 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:48.786305    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="15409c13-7939-449c-8115-31e8fc4bddf1" containerName="revoke-keys"
Jul 15 21:17:48.786516 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:48.786318    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bcdc6d6-3433-424b-b134-9a63e3df4bb4" containerName="maas-api"
Jul 15 21:17:48.786516 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:48.786326    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bcdc6d6-3433-424b-b134-9a63e3df4bb4" containerName="maas-api"
Jul 15 21:17:48.786516 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:48.786338    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc0cedd8-e570-4a06-b010-a1603c763de3" containerName="istio-proxy"
Jul 15 21:17:48.786516 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:48.786346    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc0cedd8-e570-4a06-b010-a1603c763de3" containerName="istio-proxy"
Jul 15 21:17:48.786516 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:48.786356    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab0b03e8-299b-4e2a-a227-ba1d011fac09" containerName="revoke-keys"
Jul 15 21:17:48.786516 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:48.786363    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab0b03e8-299b-4e2a-a227-ba1d011fac09" containerName="revoke-keys"
Jul 15 21:17:48.786516 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:48.786381    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc0cedd8-e570-4a06-b010-a1603c763de3" containerName="istio-proxy"
Jul 15 21:17:48.786516 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:48.786389    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc0cedd8-e570-4a06-b010-a1603c763de3" containerName="istio-proxy"
Jul 15 21:17:48.786516 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:48.786508    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bcdc6d6-3433-424b-b134-9a63e3df4bb4" containerName="maas-api"
Jul 15 21:17:48.786516 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:48.786520    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab0b03e8-299b-4e2a-a227-ba1d011fac09" containerName="revoke-keys"
Jul 15 21:17:48.787024 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:48.786532    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc0cedd8-e570-4a06-b010-a1603c763de3" containerName="istio-proxy"
Jul 15 21:17:48.787024 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:48.786541    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc0cedd8-e570-4a06-b010-a1603c763de3" containerName="istio-proxy"
Jul 15 21:17:48.789497 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:48.789475    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk"
Jul 15 21:17:48.792487 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:48.792466    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 15 21:17:48.792581 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:48.792465    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-af79a2b5-openshift-default-dockercfg-24pxt\""
Jul 15 21:17:48.801039 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:48.801016    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk"]
Jul 15 21:17:48.821609 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:48.821577    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:17:48.948670 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:48.948636    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-workload-certs\") pod \"e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk\" (UID: \"8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4\") " pod="openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk"
Jul 15 21:17:48.948849 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:48.948679    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-credential-socket\") pod \"e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk\" (UID: \"8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4\") " pod="openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk"
Jul 15 21:17:48.948849 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:48.948747    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-istio-data\") pod \"e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk\" (UID: \"8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4\") " pod="openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk"
Jul 15 21:17:48.948849 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:48.948781    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-istio-envoy\") pod \"e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk\" (UID: \"8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4\") " pod="openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk"
Jul 15 21:17:48.948849 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:48.948817    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-istio-podinfo\") pod \"e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk\" (UID: \"8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4\") " pod="openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk"
Jul 15 21:17:48.949046 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:48.948881    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-istiod-ca-cert\") pod \"e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk\" (UID: \"8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4\") " pod="openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk"
Jul 15 21:17:48.949046 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:48.948935    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-workload-socket\") pod \"e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk\" (UID: \"8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4\") " pod="openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk"
Jul 15 21:17:48.949046 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:48.948953    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc8ww\" (UniqueName: \"kubernetes.io/projected/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-kube-api-access-cc8ww\") pod \"e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk\" (UID: \"8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4\") " pod="openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk"
Jul 15 21:17:48.949046 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:48.949002    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-istio-token\") pod \"e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk\" (UID: \"8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4\") " pod="openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk"
Jul 15 21:17:49.049544 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:49.049510    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-istiod-ca-cert\") pod \"e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk\" (UID: \"8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4\") " pod="openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk"
Jul 15 21:17:49.049544 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:49.049550    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-workload-socket\") pod \"e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk\" (UID: \"8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4\") " pod="openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk"
Jul 15 21:17:49.049801 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:49.049566    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc8ww\" (UniqueName: \"kubernetes.io/projected/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-kube-api-access-cc8ww\") pod \"e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk\" (UID: \"8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4\") " pod="openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk"
Jul 15 21:17:49.049801 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:49.049598    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-istio-token\") pod \"e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk\" (UID: \"8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4\") " pod="openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk"
Jul 15 21:17:49.049801 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:49.049647    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-workload-certs\") pod \"e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk\" (UID: \"8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4\") " pod="openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk"
Jul 15 21:17:49.049801 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:49.049665    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-credential-socket\") pod \"e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk\" (UID: \"8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4\") " pod="openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk"
Jul 15 21:17:49.049801 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:49.049681    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-istio-data\") pod \"e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk\" (UID: \"8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4\") " pod="openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk"
Jul 15 21:17:49.049801 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:49.049706    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-istio-envoy\") pod \"e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk\" (UID: \"8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4\") " pod="openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk"
Jul 15 21:17:49.049801 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:49.049731    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-istio-podinfo\") pod \"e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk\" (UID: \"8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4\") " pod="openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk"
Jul 15 21:17:49.050199 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:49.050074    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-workload-socket\") pod \"e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk\" (UID: \"8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4\") " pod="openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk"
Jul 15 21:17:49.050199 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:49.050136    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-workload-certs\") pod \"e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk\" (UID: \"8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4\") " pod="openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk"
Jul 15 21:17:49.050199 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:49.050170    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-credential-socket\") pod \"e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk\" (UID: \"8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4\") " pod="openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk"
Jul 15 21:17:49.050365 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:49.050229    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-istio-data\") pod \"e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk\" (UID: \"8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4\") " pod="openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk"
Jul 15 21:17:49.050436 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:49.050373    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-istiod-ca-cert\") pod \"e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk\" (UID: \"8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4\") " pod="openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk"
Jul 15 21:17:49.052102 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:49.052075    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-istio-envoy\") pod \"e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk\" (UID: \"8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4\") " pod="openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk"
Jul 15 21:17:49.052247 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:49.052232    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-istio-podinfo\") pod \"e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk\" (UID: \"8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4\") " pod="openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk"
Jul 15 21:17:49.066354 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:49.066312    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc8ww\" (UniqueName: \"kubernetes.io/projected/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-kube-api-access-cc8ww\") pod \"e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk\" (UID: \"8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4\") " pod="openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk"
Jul 15 21:17:49.066769 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:49.066745    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-istio-token\") pod \"e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk\" (UID: \"8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4\") " pod="openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk"
Jul 15 21:17:49.100711 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:49.100678    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk"
Jul 15 21:17:49.240660 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:49.240502    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk"]
Jul 15 21:17:49.242813 ip-10-0-137-234 kubenswrapper[2566]: W0715 21:17:49.242786    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f3c091d_d2cb_43aa_bd7b_a25777c8a2c4.slice/crio-a0227cfa6de59c532eea7471d5eba18807d2fe2445cf764dc78ce391780c5e7a WatchSource:0}: Error finding container a0227cfa6de59c532eea7471d5eba18807d2fe2445cf764dc78ce391780c5e7a: Status 404 returned error can't find the container with id a0227cfa6de59c532eea7471d5eba18807d2fe2445cf764dc78ce391780c5e7a
Jul 15 21:17:49.244922 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:49.244871    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:17:49.245008 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:49.244966    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:17:49.245008 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:49.245005    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:17:49.603725 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:49.603620    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk" event={"ID":"8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4","Type":"ContainerStarted","Data":"0ee137a11412421adae02150dce1554c08a98ed6a83a914015e47a7b61f7f42b"}
Jul 15 21:17:49.603725 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:49.603685    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk" event={"ID":"8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4","Type":"ContainerStarted","Data":"a0227cfa6de59c532eea7471d5eba18807d2fe2445cf764dc78ce391780c5e7a"}
Jul 15 21:17:49.628513 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:49.628460    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk" podStartSLOduration=1.6284432290000002 podStartE2EDuration="1.628443229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:17:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:17:49.627683348 +0000 UTC m=+1802.776721406" watchObservedRunningTime="2026-07-15 21:17:49.628443229 +0000 UTC m=+1802.777481286"
Jul 15 21:17:49.756396 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:49.756363    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc0cedd8-e570-4a06-b010-a1603c763de3" path="/var/lib/kubelet/pods/dc0cedd8-e570-4a06-b010-a1603c763de3/volumes"
Jul 15 21:17:49.959014 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:49.958983    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:17:50.043857 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:50.043821    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-af79a2b5-79559dcb58-r4n2w"]
Jul 15 21:17:50.047926 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:50.047887    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af79a2b5-79559dcb58-r4n2w"
Jul 15 21:17:50.051526 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:50.051482    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-af79a2b5-serving-cert\""
Jul 15 21:17:50.051526 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:50.051515    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 15 21:17:50.051754 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:50.051551    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 15 21:17:50.051754 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:50.051590    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-s2l9d\""
Jul 15 21:17:50.058889 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:50.058786    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:17:50.060627 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:50.060603    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-af79a2b5-79559dcb58-r4n2w"]
Jul 15 21:17:50.101117 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:50.101088    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk"
Jul 15 21:17:50.106493 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:50.106466    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk"
Jul 15 21:17:50.148261 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:50.148212    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:17:50.164346 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:50.164318    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-899mg\" (UniqueName: \"kubernetes.io/projected/a50074f8-6337-47b9-861f-6d2e5f25b264-kube-api-access-899mg\") pod \"maas-api-e2e-ait-af79a2b5-79559dcb58-r4n2w\" (UID: \"a50074f8-6337-47b9-861f-6d2e5f25b264\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-af79a2b5-79559dcb58-r4n2w"
Jul 15 21:17:50.164509 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:50.164368    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a50074f8-6337-47b9-861f-6d2e5f25b264-openshift-service-ca\") pod \"maas-api-e2e-ait-af79a2b5-79559dcb58-r4n2w\" (UID: \"a50074f8-6337-47b9-861f-6d2e5f25b264\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-af79a2b5-79559dcb58-r4n2w"
Jul 15 21:17:50.164509 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:50.164392    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a50074f8-6337-47b9-861f-6d2e5f25b264-maas-api-tls\") pod \"maas-api-e2e-ait-af79a2b5-79559dcb58-r4n2w\" (UID: \"a50074f8-6337-47b9-861f-6d2e5f25b264\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-af79a2b5-79559dcb58-r4n2w"
Jul 15 21:17:50.265773 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:50.265694    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-899mg\" (UniqueName: \"kubernetes.io/projected/a50074f8-6337-47b9-861f-6d2e5f25b264-kube-api-access-899mg\") pod \"maas-api-e2e-ait-af79a2b5-79559dcb58-r4n2w\" (UID: \"a50074f8-6337-47b9-861f-6d2e5f25b264\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-af79a2b5-79559dcb58-r4n2w"
Jul 15 21:17:50.265773 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:50.265747    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a50074f8-6337-47b9-861f-6d2e5f25b264-openshift-service-ca\") pod \"maas-api-e2e-ait-af79a2b5-79559dcb58-r4n2w\" (UID: \"a50074f8-6337-47b9-861f-6d2e5f25b264\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-af79a2b5-79559dcb58-r4n2w"
Jul 15 21:17:50.265773 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:50.265774    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a50074f8-6337-47b9-861f-6d2e5f25b264-maas-api-tls\") pod \"maas-api-e2e-ait-af79a2b5-79559dcb58-r4n2w\" (UID: \"a50074f8-6337-47b9-861f-6d2e5f25b264\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-af79a2b5-79559dcb58-r4n2w"
Jul 15 21:17:50.266036 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:17:50.265873    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-af79a2b5-serving-cert: secret "maas-api-e2e-ait-af79a2b5-serving-cert" not found
Jul 15 21:17:50.266036 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:17:50.265970    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a50074f8-6337-47b9-861f-6d2e5f25b264-maas-api-tls podName:a50074f8-6337-47b9-861f-6d2e5f25b264 nodeName:}" failed. No retries permitted until 2026-07-15 21:17:50.765950143 +0000 UTC m=+1803.914988196 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a50074f8-6337-47b9-861f-6d2e5f25b264-maas-api-tls") pod "maas-api-e2e-ait-af79a2b5-79559dcb58-r4n2w" (UID: "a50074f8-6337-47b9-861f-6d2e5f25b264") : secret "maas-api-e2e-ait-af79a2b5-serving-cert" not found
Jul 15 21:17:50.266520 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:50.266493    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a50074f8-6337-47b9-861f-6d2e5f25b264-openshift-service-ca\") pod \"maas-api-e2e-ait-af79a2b5-79559dcb58-r4n2w\" (UID: \"a50074f8-6337-47b9-861f-6d2e5f25b264\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-af79a2b5-79559dcb58-r4n2w"
Jul 15 21:17:50.277711 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:50.277684    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-899mg\" (UniqueName: \"kubernetes.io/projected/a50074f8-6337-47b9-861f-6d2e5f25b264-kube-api-access-899mg\") pod \"maas-api-e2e-ait-af79a2b5-79559dcb58-r4n2w\" (UID: \"a50074f8-6337-47b9-861f-6d2e5f25b264\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-af79a2b5-79559dcb58-r4n2w"
Jul 15 21:17:50.610951 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:50.610846    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk"
Jul 15 21:17:50.611872 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:50.611852    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk"
Jul 15 21:17:50.662388 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:50.662331    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:17:50.759251 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:50.759218    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:17:50.770955 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:50.770927    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a50074f8-6337-47b9-861f-6d2e5f25b264-maas-api-tls\") pod \"maas-api-e2e-ait-af79a2b5-79559dcb58-r4n2w\" (UID: \"a50074f8-6337-47b9-861f-6d2e5f25b264\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-af79a2b5-79559dcb58-r4n2w"
Jul 15 21:17:50.771096 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:17:50.771078    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-af79a2b5-serving-cert: secret "maas-api-e2e-ait-af79a2b5-serving-cert" not found
Jul 15 21:17:50.771180 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:17:50.771165    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a50074f8-6337-47b9-861f-6d2e5f25b264-maas-api-tls podName:a50074f8-6337-47b9-861f-6d2e5f25b264 nodeName:}" failed. No retries permitted until 2026-07-15 21:17:51.771144254 +0000 UTC m=+1804.920182289 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a50074f8-6337-47b9-861f-6d2e5f25b264-maas-api-tls") pod "maas-api-e2e-ait-af79a2b5-79559dcb58-r4n2w" (UID: "a50074f8-6337-47b9-861f-6d2e5f25b264") : secret "maas-api-e2e-ait-af79a2b5-serving-cert" not found
Jul 15 21:17:51.011253 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:51.011216    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:17:51.782136 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:51.782089    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a50074f8-6337-47b9-861f-6d2e5f25b264-maas-api-tls\") pod \"maas-api-e2e-ait-af79a2b5-79559dcb58-r4n2w\" (UID: \"a50074f8-6337-47b9-861f-6d2e5f25b264\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-af79a2b5-79559dcb58-r4n2w"
Jul 15 21:17:51.785551 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:51.785523    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a50074f8-6337-47b9-861f-6d2e5f25b264-maas-api-tls\") pod \"maas-api-e2e-ait-af79a2b5-79559dcb58-r4n2w\" (UID: \"a50074f8-6337-47b9-861f-6d2e5f25b264\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-af79a2b5-79559dcb58-r4n2w"
Jul 15 21:17:51.869319 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:51.869282    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af79a2b5-79559dcb58-r4n2w"
Jul 15 21:17:52.005272 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:52.005246    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-af79a2b5-79559dcb58-r4n2w"]
Jul 15 21:17:52.006791 ip-10-0-137-234 kubenswrapper[2566]: W0715 21:17:52.006767    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda50074f8_6337_47b9_861f_6d2e5f25b264.slice/crio-4f5276338ec27991f19e4af8e1773f9e623c082a3c0af2edc8a6c962e002ea9c WatchSource:0}: Error finding container 4f5276338ec27991f19e4af8e1773f9e623c082a3c0af2edc8a6c962e002ea9c: Status 404 returned error can't find the container with id 4f5276338ec27991f19e4af8e1773f9e623c082a3c0af2edc8a6c962e002ea9c
Jul 15 21:17:52.621825 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:52.621787    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af79a2b5-79559dcb58-r4n2w" event={"ID":"a50074f8-6337-47b9-861f-6d2e5f25b264","Type":"ContainerStarted","Data":"6009df7760385c11c7b27e33c6bb22326b1c4a548387ba3ab8d34fb6e807c85f"}
Jul 15 21:17:52.622046 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:52.622026    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af79a2b5-79559dcb58-r4n2w"
Jul 15 21:17:52.622116 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:52.622068    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af79a2b5-79559dcb58-r4n2w" event={"ID":"a50074f8-6337-47b9-861f-6d2e5f25b264","Type":"ContainerStarted","Data":"4f5276338ec27991f19e4af8e1773f9e623c082a3c0af2edc8a6c962e002ea9c"}
Jul 15 21:17:52.641025 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:52.640969    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af79a2b5-79559dcb58-r4n2w" podStartSLOduration=2.203424398 podStartE2EDuration="2.640952048s" totalImagesPullingTime="437.52765ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:17:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:17:52.640466548 +0000 UTC m=+1805.789504605" watchObservedRunningTime="2026-07-15 21:17:52.640952048 +0000 UTC m=+1805.789990118"
Jul 15 21:17:53.724251 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:53.724216    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:17:53.825226 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:53.825186    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:17:55.169608 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:55.169566    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af79a2b5-z7mwz"]
Jul 15 21:17:55.173273 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:55.173253    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af79a2b5-z7mwz"
Jul 15 21:17:55.176245 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:55.176220    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zsrf5\""
Jul 15 21:17:55.183103 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:55.183067    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af79a2b5-z7mwz"]
Jul 15 21:17:55.313541 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:55.313506    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f32faca6-4dd7-4d56-923a-d9c747264430-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af79a2b5-z7mwz\" (UID: \"f32faca6-4dd7-4d56-923a-d9c747264430\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af79a2b5-z7mwz"
Jul 15 21:17:55.414873 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:55.414834    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f32faca6-4dd7-4d56-923a-d9c747264430-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af79a2b5-z7mwz\" (UID: \"f32faca6-4dd7-4d56-923a-d9c747264430\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af79a2b5-z7mwz"
Jul 15 21:17:55.415514 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:55.415493    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f32faca6-4dd7-4d56-923a-d9c747264430-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af79a2b5-z7mwz\" (UID: \"f32faca6-4dd7-4d56-923a-d9c747264430\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af79a2b5-z7mwz"
Jul 15 21:17:55.485415 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:55.485383    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af79a2b5-z7mwz"
Jul 15 21:17:55.594740 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:55.594699    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:17:55.701582 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:55.701548    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:17:55.829481 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:55.829456    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af79a2b5-z7mwz"]
Jul 15 21:17:55.832130 ip-10-0-137-234 kubenswrapper[2566]: W0715 21:17:55.832099    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf32faca6_4dd7_4d56_923a_d9c747264430.slice/crio-3d46bbd21bb4f14536d68720cd6e94dfa43a2b1a309edc01c793657ed3627510 WatchSource:0}: Error finding container 3d46bbd21bb4f14536d68720cd6e94dfa43a2b1a309edc01c793657ed3627510: Status 404 returned error can't find the container with id 3d46bbd21bb4f14536d68720cd6e94dfa43a2b1a309edc01c793657ed3627510
Jul 15 21:17:56.473742 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:56.473707    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:17:56.585265 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:56.585181    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:17:56.644565 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:56.644531    2566 generic.go:356] "Generic (PLEG): container finished" podID="f32faca6-4dd7-4d56-923a-d9c747264430" containerID="2fd02c67c4bc1fdafd938c7c9f7a8bb58b678adfbd5dc9713828479b122f54a1" exitCode=7
Jul 15 21:17:56.644730 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:56.644572    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af79a2b5-z7mwz" event={"ID":"f32faca6-4dd7-4d56-923a-d9c747264430","Type":"ContainerDied","Data":"2fd02c67c4bc1fdafd938c7c9f7a8bb58b678adfbd5dc9713828479b122f54a1"}
Jul 15 21:17:56.644730 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:56.644609    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af79a2b5-z7mwz" event={"ID":"f32faca6-4dd7-4d56-923a-d9c747264430","Type":"ContainerStarted","Data":"3d46bbd21bb4f14536d68720cd6e94dfa43a2b1a309edc01c793657ed3627510"}
Jul 15 21:17:56.644816 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:56.644778    2566 scope.go:117] "RemoveContainer" containerID="2fd02c67c4bc1fdafd938c7c9f7a8bb58b678adfbd5dc9713828479b122f54a1"
Jul 15 21:17:57.515242 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:57.515209    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:17:57.652066 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:57.651978    2566 generic.go:356] "Generic (PLEG): container finished" podID="f32faca6-4dd7-4d56-923a-d9c747264430" containerID="efab4c8a50f0fcc5b621cd11387405a62441b392803dfd616bfc260d920f2d49" exitCode=7
Jul 15 21:17:57.652066 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:57.652047    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af79a2b5-z7mwz" event={"ID":"f32faca6-4dd7-4d56-923a-d9c747264430","Type":"ContainerDied","Data":"efab4c8a50f0fcc5b621cd11387405a62441b392803dfd616bfc260d920f2d49"}
Jul 15 21:17:57.652250 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:57.652105    2566 scope.go:117] "RemoveContainer" containerID="2fd02c67c4bc1fdafd938c7c9f7a8bb58b678adfbd5dc9713828479b122f54a1"
Jul 15 21:17:57.652309 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:57.652295    2566 scope.go:117] "RemoveContainer" containerID="efab4c8a50f0fcc5b621cd11387405a62441b392803dfd616bfc260d920f2d49"
Jul 15 21:17:57.652524 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:17:57.652506    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af79a2b5-z7mwz_odh-ai-gateway-infra(f32faca6-4dd7-4d56-923a-d9c747264430)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af79a2b5-z7mwz" podUID="f32faca6-4dd7-4d56-923a-d9c747264430"
Jul 15 21:17:58.632047 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:58.632011    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af79a2b5-79559dcb58-r4n2w"
Jul 15 21:17:58.658875 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:17:58.658851    2566 scope.go:117] "RemoveContainer" containerID="efab4c8a50f0fcc5b621cd11387405a62441b392803dfd616bfc260d920f2d49"
Jul 15 21:17:58.659068 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:17:58.659048    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af79a2b5-z7mwz_odh-ai-gateway-infra(f32faca6-4dd7-4d56-923a-d9c747264430)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af79a2b5-z7mwz" podUID="f32faca6-4dd7-4d56-923a-d9c747264430"
Jul 15 21:18:06.402234 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:06.402198    2566 patch_prober.go:28] interesting pod/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": read tcp 10.134.0.2:54320->10.134.0.41:15021: read: connection reset by peer" start-of-body=
Jul 15 21:18:06.402664 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:06.402280    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk" podUID="8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": read tcp 10.134.0.2:54320->10.134.0.41:15021: read: connection reset by peer"
Jul 15 21:18:06.703259 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:06.703225    2566 generic.go:356] "Generic (PLEG): container finished" podID="8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4" containerID="0ee137a11412421adae02150dce1554c08a98ed6a83a914015e47a7b61f7f42b" exitCode=0
Jul 15 21:18:06.703420 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:06.703301    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk" event={"ID":"8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4","Type":"ContainerDied","Data":"0ee137a11412421adae02150dce1554c08a98ed6a83a914015e47a7b61f7f42b"}
Jul 15 21:18:06.703722 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:06.703708    2566 scope.go:117] "RemoveContainer" containerID="0ee137a11412421adae02150dce1554c08a98ed6a83a914015e47a7b61f7f42b"
Jul 15 21:18:06.704781 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:06.704758    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:18:06.704842 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:06.704828    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:18:06.704880 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:06.704861    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:18:07.712160 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:07.712119    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk" event={"ID":"8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4","Type":"ContainerStarted","Data":"5c59798682e8ded9e13617de8622125bba3efdda8b52edd68e20112819a12fac"}
Jul 15 21:18:08.101043 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:08.100876    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk"
Jul 15 21:18:08.101254 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:08.101223    2566 patch_prober.go:28] interesting pod/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Jul 15 21:18:08.101472 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:08.101442    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk" podUID="8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Jul 15 21:18:09.101623 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:09.101585    2566 patch_prober.go:28] interesting pod/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Jul 15 21:18:09.102249 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:09.102209    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk" podUID="8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Jul 15 21:18:10.101999 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:10.101959    2566 patch_prober.go:28] interesting pod/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Jul 15 21:18:10.102501 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:10.102040    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk" podUID="8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Jul 15 21:18:11.102191 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:11.102146    2566 patch_prober.go:28] interesting pod/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Jul 15 21:18:11.102731 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:11.102213    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk" podUID="8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Jul 15 21:18:12.104616 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:12.104588    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk"
Jul 15 21:18:12.105029 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:12.104841    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk"
Jul 15 21:18:12.105640 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:12.105620    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk"
Jul 15 21:18:13.751563 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:13.751532    2566 scope.go:117] "RemoveContainer" containerID="efab4c8a50f0fcc5b621cd11387405a62441b392803dfd616bfc260d920f2d49"
Jul 15 21:18:14.751688 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:14.751655    2566 generic.go:356] "Generic (PLEG): container finished" podID="f32faca6-4dd7-4d56-923a-d9c747264430" containerID="e63a87ffe99163aaf02d496ace3dc3f9f9b349c8078f10e6d9a1f9a56e2095a1" exitCode=0
Jul 15 21:18:14.752168 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:14.751718    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af79a2b5-z7mwz" event={"ID":"f32faca6-4dd7-4d56-923a-d9c747264430","Type":"ContainerDied","Data":"e63a87ffe99163aaf02d496ace3dc3f9f9b349c8078f10e6d9a1f9a56e2095a1"}
Jul 15 21:18:14.752168 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:14.751816    2566 scope.go:117] "RemoveContainer" containerID="efab4c8a50f0fcc5b621cd11387405a62441b392803dfd616bfc260d920f2d49"
Jul 15 21:18:15.785233 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:15.785195    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af79a2b5-z7mwz"]
Jul 15 21:18:15.889805 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:15.889779    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af79a2b5-z7mwz"
Jul 15 21:18:16.006655 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:16.006619    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f32faca6-4dd7-4d56-923a-d9c747264430-maas-api-service-ca\") pod \"f32faca6-4dd7-4d56-923a-d9c747264430\" (UID: \"f32faca6-4dd7-4d56-923a-d9c747264430\") "
Jul 15 21:18:16.007011 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:16.006982    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f32faca6-4dd7-4d56-923a-d9c747264430-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f32faca6-4dd7-4d56-923a-d9c747264430" (UID: "f32faca6-4dd7-4d56-923a-d9c747264430"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 21:18:16.107619 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:16.107530    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f32faca6-4dd7-4d56-923a-d9c747264430-maas-api-service-ca\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:18:16.764522 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:16.764491    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af79a2b5-z7mwz"
Jul 15 21:18:16.764708 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:16.764486    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af79a2b5-z7mwz" event={"ID":"f32faca6-4dd7-4d56-923a-d9c747264430","Type":"ContainerDied","Data":"3d46bbd21bb4f14536d68720cd6e94dfa43a2b1a309edc01c793657ed3627510"}
Jul 15 21:18:16.764708 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:16.764615    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d46bbd21bb4f14536d68720cd6e94dfa43a2b1a309edc01c793657ed3627510"
Jul 15 21:18:16.811986 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:16.811953    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af79a2b5-z7mwz"]
Jul 15 21:18:16.826619 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:16.826586    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af79a2b5-z7mwz"]
Jul 15 21:18:17.756984 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:17.756951    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f32faca6-4dd7-4d56-923a-d9c747264430" path="/var/lib/kubelet/pods/f32faca6-4dd7-4d56-923a-d9c747264430/volumes"
Jul 15 21:18:55.245753 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:55.245707    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af79a2b5-qvwzc"]
Jul 15 21:18:55.246205 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:55.246073    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f32faca6-4dd7-4d56-923a-d9c747264430" containerName="revoke-keys"
Jul 15 21:18:55.246205 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:55.246085    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f32faca6-4dd7-4d56-923a-d9c747264430" containerName="revoke-keys"
Jul 15 21:18:55.246205 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:55.246094    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f32faca6-4dd7-4d56-923a-d9c747264430" containerName="revoke-keys"
Jul 15 21:18:55.246205 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:55.246101    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f32faca6-4dd7-4d56-923a-d9c747264430" containerName="revoke-keys"
Jul 15 21:18:55.246205 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:55.246162    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f32faca6-4dd7-4d56-923a-d9c747264430" containerName="revoke-keys"
Jul 15 21:18:55.246205 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:55.246170    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f32faca6-4dd7-4d56-923a-d9c747264430" containerName="revoke-keys"
Jul 15 21:18:55.248792 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:55.248775    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af79a2b5-qvwzc"
Jul 15 21:18:55.251825 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:55.251798    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zsrf5\""
Jul 15 21:18:55.256985 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:55.256862    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af79a2b5-qvwzc"]
Jul 15 21:18:55.334967 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:55.334931    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ef064e5-f395-44e5-a27e-788ab15459b1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af79a2b5-qvwzc\" (UID: \"4ef064e5-f395-44e5-a27e-788ab15459b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af79a2b5-qvwzc"
Jul 15 21:18:55.436314 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:55.436281    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ef064e5-f395-44e5-a27e-788ab15459b1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af79a2b5-qvwzc\" (UID: \"4ef064e5-f395-44e5-a27e-788ab15459b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af79a2b5-qvwzc"
Jul 15 21:18:55.436966 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:55.436936    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ef064e5-f395-44e5-a27e-788ab15459b1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af79a2b5-qvwzc\" (UID: \"4ef064e5-f395-44e5-a27e-788ab15459b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af79a2b5-qvwzc"
Jul 15 21:18:55.562081 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:55.561989    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af79a2b5-qvwzc"
Jul 15 21:18:55.699379 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:55.699313    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af79a2b5-qvwzc"]
Jul 15 21:18:55.701673 ip-10-0-137-234 kubenswrapper[2566]: W0715 21:18:55.701642    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ef064e5_f395_44e5_a27e_788ab15459b1.slice/crio-46071f99e1bb356dc4c326a13a664bcc40397ec8819cb6b596d82f1715006d5f WatchSource:0}: Error finding container 46071f99e1bb356dc4c326a13a664bcc40397ec8819cb6b596d82f1715006d5f: Status 404 returned error can't find the container with id 46071f99e1bb356dc4c326a13a664bcc40397ec8819cb6b596d82f1715006d5f
Jul 15 21:18:55.950246 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:55.950210    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af79a2b5-qvwzc" event={"ID":"4ef064e5-f395-44e5-a27e-788ab15459b1","Type":"ContainerStarted","Data":"46071f99e1bb356dc4c326a13a664bcc40397ec8819cb6b596d82f1715006d5f"}
Jul 15 21:18:56.956956 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:56.956917    2566 generic.go:356] "Generic (PLEG): container finished" podID="4ef064e5-f395-44e5-a27e-788ab15459b1" containerID="e70fca3bd401fc3166a23f88673a1ea0855bff7926181d1020e5f59ff9fdf9f0" exitCode=0
Jul 15 21:18:56.957332 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:56.957006    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af79a2b5-qvwzc" event={"ID":"4ef064e5-f395-44e5-a27e-788ab15459b1","Type":"ContainerDied","Data":"e70fca3bd401fc3166a23f88673a1ea0855bff7926181d1020e5f59ff9fdf9f0"}
Jul 15 21:18:58.097675 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:58.097649    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af79a2b5-qvwzc"
Jul 15 21:18:58.161306 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:58.161272    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ef064e5-f395-44e5-a27e-788ab15459b1-maas-api-service-ca\") pod \"4ef064e5-f395-44e5-a27e-788ab15459b1\" (UID: \"4ef064e5-f395-44e5-a27e-788ab15459b1\") "
Jul 15 21:18:58.161667 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:58.161642    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ef064e5-f395-44e5-a27e-788ab15459b1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4ef064e5-f395-44e5-a27e-788ab15459b1" (UID: "4ef064e5-f395-44e5-a27e-788ab15459b1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 21:18:58.261859 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:58.261768    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ef064e5-f395-44e5-a27e-788ab15459b1-maas-api-service-ca\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:18:58.968502 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:58.968476    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af79a2b5-qvwzc"
Jul 15 21:18:58.968679 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:58.968467    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af79a2b5-qvwzc" event={"ID":"4ef064e5-f395-44e5-a27e-788ab15459b1","Type":"ContainerDied","Data":"46071f99e1bb356dc4c326a13a664bcc40397ec8819cb6b596d82f1715006d5f"}
Jul 15 21:18:58.968679 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:18:58.968609    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46071f99e1bb356dc4c326a13a664bcc40397ec8819cb6b596d82f1715006d5f"
Jul 15 21:19:05.707585 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:05.707515    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-af79a2b5-79559dcb58-r4n2w"]
Jul 15 21:19:05.708395 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:05.708356    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af79a2b5-79559dcb58-r4n2w" podUID="a50074f8-6337-47b9-861f-6d2e5f25b264" containerName="maas-api" containerID="cri-o://6009df7760385c11c7b27e33c6bb22326b1c4a548387ba3ab8d34fb6e807c85f" gracePeriod=30
Jul 15 21:19:05.731345 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:05.731308    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:19:05.962506 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:05.962434    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af79a2b5-79559dcb58-r4n2w"
Jul 15 21:19:06.002469 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:06.002438    2566 generic.go:356] "Generic (PLEG): container finished" podID="a50074f8-6337-47b9-861f-6d2e5f25b264" containerID="6009df7760385c11c7b27e33c6bb22326b1c4a548387ba3ab8d34fb6e807c85f" exitCode=0
Jul 15 21:19:06.002614 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:06.002487    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af79a2b5-79559dcb58-r4n2w"
Jul 15 21:19:06.002614 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:06.002528    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af79a2b5-79559dcb58-r4n2w" event={"ID":"a50074f8-6337-47b9-861f-6d2e5f25b264","Type":"ContainerDied","Data":"6009df7760385c11c7b27e33c6bb22326b1c4a548387ba3ab8d34fb6e807c85f"}
Jul 15 21:19:06.002614 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:06.002603    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-af79a2b5-79559dcb58-r4n2w" event={"ID":"a50074f8-6337-47b9-861f-6d2e5f25b264","Type":"ContainerDied","Data":"4f5276338ec27991f19e4af8e1773f9e623c082a3c0af2edc8a6c962e002ea9c"}
Jul 15 21:19:06.002712 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:06.002640    2566 scope.go:117] "RemoveContainer" containerID="6009df7760385c11c7b27e33c6bb22326b1c4a548387ba3ab8d34fb6e807c85f"
Jul 15 21:19:06.011289 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:06.011266    2566 scope.go:117] "RemoveContainer" containerID="6009df7760385c11c7b27e33c6bb22326b1c4a548387ba3ab8d34fb6e807c85f"
Jul 15 21:19:06.011595 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:19:06.011571    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6009df7760385c11c7b27e33c6bb22326b1c4a548387ba3ab8d34fb6e807c85f\": container with ID starting with 6009df7760385c11c7b27e33c6bb22326b1c4a548387ba3ab8d34fb6e807c85f not found: ID does not exist" containerID="6009df7760385c11c7b27e33c6bb22326b1c4a548387ba3ab8d34fb6e807c85f"
Jul 15 21:19:06.011668 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:06.011606    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6009df7760385c11c7b27e33c6bb22326b1c4a548387ba3ab8d34fb6e807c85f"} err="failed to get container status \"6009df7760385c11c7b27e33c6bb22326b1c4a548387ba3ab8d34fb6e807c85f\": rpc error: code = NotFound desc = could not find container \"6009df7760385c11c7b27e33c6bb22326b1c4a548387ba3ab8d34fb6e807c85f\": container with ID starting with 6009df7760385c11c7b27e33c6bb22326b1c4a548387ba3ab8d34fb6e807c85f not found: ID does not exist"
Jul 15 21:19:06.032991 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:06.032961    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a50074f8-6337-47b9-861f-6d2e5f25b264-openshift-service-ca\") pod \"a50074f8-6337-47b9-861f-6d2e5f25b264\" (UID: \"a50074f8-6337-47b9-861f-6d2e5f25b264\") "
Jul 15 21:19:06.033110 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:06.033004    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a50074f8-6337-47b9-861f-6d2e5f25b264-maas-api-tls\") pod \"a50074f8-6337-47b9-861f-6d2e5f25b264\" (UID: \"a50074f8-6337-47b9-861f-6d2e5f25b264\") "
Jul 15 21:19:06.033110 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:06.033061    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-899mg\" (UniqueName: \"kubernetes.io/projected/a50074f8-6337-47b9-861f-6d2e5f25b264-kube-api-access-899mg\") pod \"a50074f8-6337-47b9-861f-6d2e5f25b264\" (UID: \"a50074f8-6337-47b9-861f-6d2e5f25b264\") "
Jul 15 21:19:06.033361 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:06.033335    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a50074f8-6337-47b9-861f-6d2e5f25b264-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a50074f8-6337-47b9-861f-6d2e5f25b264" (UID: "a50074f8-6337-47b9-861f-6d2e5f25b264"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 21:19:06.034876 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:06.034846    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a50074f8-6337-47b9-861f-6d2e5f25b264-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a50074f8-6337-47b9-861f-6d2e5f25b264" (UID: "a50074f8-6337-47b9-861f-6d2e5f25b264"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 21:19:06.035061 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:06.035035    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a50074f8-6337-47b9-861f-6d2e5f25b264-kube-api-access-899mg" (OuterVolumeSpecName: "kube-api-access-899mg") pod "a50074f8-6337-47b9-861f-6d2e5f25b264" (UID: "a50074f8-6337-47b9-861f-6d2e5f25b264"). InnerVolumeSpecName "kube-api-access-899mg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 21:19:06.134460 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:06.134402    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-899mg\" (UniqueName: \"kubernetes.io/projected/a50074f8-6337-47b9-861f-6d2e5f25b264-kube-api-access-899mg\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:19:06.134460 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:06.134455    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a50074f8-6337-47b9-861f-6d2e5f25b264-openshift-service-ca\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:19:06.134460 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:06.134469    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a50074f8-6337-47b9-861f-6d2e5f25b264-maas-api-tls\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:19:06.325681 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:06.325645    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-af79a2b5-79559dcb58-r4n2w"]
Jul 15 21:19:06.329868 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:06.329840    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-af79a2b5-79559dcb58-r4n2w"]
Jul 15 21:19:07.757374 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:07.757339    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a50074f8-6337-47b9-861f-6d2e5f25b264" path="/var/lib/kubelet/pods/a50074f8-6337-47b9-861f-6d2e5f25b264/volumes"
Jul 15 21:19:10.339767 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:10.339730    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:19:16.947745 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:16.947712    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:19:17.060208 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:17.060176    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:19:21.419245 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:21.419211    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk"]
Jul 15 21:19:21.419805 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:21.419440    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk" podUID="8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4" containerName="istio-proxy" containerID="cri-o://5c59798682e8ded9e13617de8622125bba3efdda8b52edd68e20112819a12fac" gracePeriod=30
Jul 15 21:19:21.443112 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:21.443080    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:19:21.757846 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:21.757821    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr"]
Jul 15 21:19:21.758188 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:21.758172    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ef064e5-f395-44e5-a27e-788ab15459b1" containerName="revoke-keys"
Jul 15 21:19:21.758233 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:21.758190    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ef064e5-f395-44e5-a27e-788ab15459b1" containerName="revoke-keys"
Jul 15 21:19:21.758233 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:21.758201    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a50074f8-6337-47b9-861f-6d2e5f25b264" containerName="maas-api"
Jul 15 21:19:21.758233 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:21.758207    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a50074f8-6337-47b9-861f-6d2e5f25b264" containerName="maas-api"
Jul 15 21:19:21.758233 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:21.758214    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f32faca6-4dd7-4d56-923a-d9c747264430" containerName="revoke-keys"
Jul 15 21:19:21.758233 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:21.758219    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f32faca6-4dd7-4d56-923a-d9c747264430" containerName="revoke-keys"
Jul 15 21:19:21.758390 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:21.758290    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f32faca6-4dd7-4d56-923a-d9c747264430" containerName="revoke-keys"
Jul 15 21:19:21.758390 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:21.758298    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ef064e5-f395-44e5-a27e-788ab15459b1" containerName="revoke-keys"
Jul 15 21:19:21.758390 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:21.758307    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a50074f8-6337-47b9-861f-6d2e5f25b264" containerName="maas-api"
Jul 15 21:19:21.761055 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:21.761037    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr"
Jul 15 21:19:21.764340 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:21.764315    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-6e3236bd-openshift-default-dockercfg-hq7g7\""
Jul 15 21:19:21.772065 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:21.772041    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr"]
Jul 15 21:19:21.784847 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:21.784812    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:19:21.885385 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:21.885343    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f22efb3-18ab-43a2-a936-4a717371ece3-istio-podinfo\") pod \"e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr\" (UID: \"4f22efb3-18ab-43a2-a936-4a717371ece3\") " pod="openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr"
Jul 15 21:19:21.885567 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:21.885403    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f22efb3-18ab-43a2-a936-4a717371ece3-istio-data\") pod \"e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr\" (UID: \"4f22efb3-18ab-43a2-a936-4a717371ece3\") " pod="openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr"
Jul 15 21:19:21.885567 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:21.885449    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f22efb3-18ab-43a2-a936-4a717371ece3-workload-socket\") pod \"e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr\" (UID: \"4f22efb3-18ab-43a2-a936-4a717371ece3\") " pod="openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr"
Jul 15 21:19:21.885567 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:21.885495    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f22efb3-18ab-43a2-a936-4a717371ece3-istiod-ca-cert\") pod \"e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr\" (UID: \"4f22efb3-18ab-43a2-a936-4a717371ece3\") " pod="openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr"
Jul 15 21:19:21.885567 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:21.885531    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f22efb3-18ab-43a2-a936-4a717371ece3-credential-socket\") pod \"e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr\" (UID: \"4f22efb3-18ab-43a2-a936-4a717371ece3\") " pod="openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr"
Jul 15 21:19:21.885567 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:21.885558    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f22efb3-18ab-43a2-a936-4a717371ece3-istio-token\") pod \"e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr\" (UID: \"4f22efb3-18ab-43a2-a936-4a717371ece3\") " pod="openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr"
Jul 15 21:19:21.885779 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:21.885584    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9cdj\" (UniqueName: \"kubernetes.io/projected/4f22efb3-18ab-43a2-a936-4a717371ece3-kube-api-access-c9cdj\") pod \"e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr\" (UID: \"4f22efb3-18ab-43a2-a936-4a717371ece3\") " pod="openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr"
Jul 15 21:19:21.885779 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:21.885620    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f22efb3-18ab-43a2-a936-4a717371ece3-istio-envoy\") pod \"e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr\" (UID: \"4f22efb3-18ab-43a2-a936-4a717371ece3\") " pod="openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr"
Jul 15 21:19:21.885779 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:21.885678    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f22efb3-18ab-43a2-a936-4a717371ece3-workload-certs\") pod \"e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr\" (UID: \"4f22efb3-18ab-43a2-a936-4a717371ece3\") " pod="openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr"
Jul 15 21:19:21.986627 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:21.986589    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f22efb3-18ab-43a2-a936-4a717371ece3-istio-envoy\") pod \"e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr\" (UID: \"4f22efb3-18ab-43a2-a936-4a717371ece3\") " pod="openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr"
Jul 15 21:19:21.986826 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:21.986637    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f22efb3-18ab-43a2-a936-4a717371ece3-workload-certs\") pod \"e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr\" (UID: \"4f22efb3-18ab-43a2-a936-4a717371ece3\") " pod="openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr"
Jul 15 21:19:21.986826 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:21.986677    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f22efb3-18ab-43a2-a936-4a717371ece3-istio-podinfo\") pod \"e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr\" (UID: \"4f22efb3-18ab-43a2-a936-4a717371ece3\") " pod="openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr"
Jul 15 21:19:21.986826 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:21.986706    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f22efb3-18ab-43a2-a936-4a717371ece3-istio-data\") pod \"e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr\" (UID: \"4f22efb3-18ab-43a2-a936-4a717371ece3\") " pod="openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr"
Jul 15 21:19:21.986826 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:21.986734    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f22efb3-18ab-43a2-a936-4a717371ece3-workload-socket\") pod \"e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr\" (UID: \"4f22efb3-18ab-43a2-a936-4a717371ece3\") " pod="openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr"
Jul 15 21:19:21.986826 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:21.986772    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f22efb3-18ab-43a2-a936-4a717371ece3-istiod-ca-cert\") pod \"e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr\" (UID: \"4f22efb3-18ab-43a2-a936-4a717371ece3\") " pod="openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr"
Jul 15 21:19:21.986826 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:21.986808    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f22efb3-18ab-43a2-a936-4a717371ece3-credential-socket\") pod \"e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr\" (UID: \"4f22efb3-18ab-43a2-a936-4a717371ece3\") " pod="openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr"
Jul 15 21:19:21.987176 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:21.986833    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f22efb3-18ab-43a2-a936-4a717371ece3-istio-token\") pod \"e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr\" (UID: \"4f22efb3-18ab-43a2-a936-4a717371ece3\") " pod="openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr"
Jul 15 21:19:21.987176 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:21.986860    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9cdj\" (UniqueName: \"kubernetes.io/projected/4f22efb3-18ab-43a2-a936-4a717371ece3-kube-api-access-c9cdj\") pod \"e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr\" (UID: \"4f22efb3-18ab-43a2-a936-4a717371ece3\") " pod="openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr"
Jul 15 21:19:21.987272 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:21.987216    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f22efb3-18ab-43a2-a936-4a717371ece3-istio-data\") pod \"e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr\" (UID: \"4f22efb3-18ab-43a2-a936-4a717371ece3\") " pod="openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr"
Jul 15 21:19:21.987331 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:21.987260    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f22efb3-18ab-43a2-a936-4a717371ece3-credential-socket\") pod \"e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr\" (UID: \"4f22efb3-18ab-43a2-a936-4a717371ece3\") " pod="openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr"
Jul 15 21:19:21.987440 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:21.987419    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f22efb3-18ab-43a2-a936-4a717371ece3-workload-socket\") pod \"e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr\" (UID: \"4f22efb3-18ab-43a2-a936-4a717371ece3\") " pod="openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr"
Jul 15 21:19:21.987635 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:21.987613    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f22efb3-18ab-43a2-a936-4a717371ece3-workload-certs\") pod \"e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr\" (UID: \"4f22efb3-18ab-43a2-a936-4a717371ece3\") " pod="openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr"
Jul 15 21:19:21.987789 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:21.987762    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f22efb3-18ab-43a2-a936-4a717371ece3-istiod-ca-cert\") pod \"e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr\" (UID: \"4f22efb3-18ab-43a2-a936-4a717371ece3\") " pod="openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr"
Jul 15 21:19:21.989144 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:21.989122    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f22efb3-18ab-43a2-a936-4a717371ece3-istio-envoy\") pod \"e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr\" (UID: \"4f22efb3-18ab-43a2-a936-4a717371ece3\") " pod="openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr"
Jul 15 21:19:21.989400 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:21.989385    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f22efb3-18ab-43a2-a936-4a717371ece3-istio-podinfo\") pod \"e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr\" (UID: \"4f22efb3-18ab-43a2-a936-4a717371ece3\") " pod="openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr"
Jul 15 21:19:21.996572 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:21.996549    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9cdj\" (UniqueName: \"kubernetes.io/projected/4f22efb3-18ab-43a2-a936-4a717371ece3-kube-api-access-c9cdj\") pod \"e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr\" (UID: \"4f22efb3-18ab-43a2-a936-4a717371ece3\") " pod="openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr"
Jul 15 21:19:21.996669 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:21.996593    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f22efb3-18ab-43a2-a936-4a717371ece3-istio-token\") pod \"e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr\" (UID: \"4f22efb3-18ab-43a2-a936-4a717371ece3\") " pod="openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr"
Jul 15 21:19:22.072099 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:22.072012    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr"
Jul 15 21:19:22.219155 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:22.219129    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr"]
Jul 15 21:19:22.221248 ip-10-0-137-234 kubenswrapper[2566]: W0715 21:19:22.221220    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f22efb3_18ab_43a2_a936_4a717371ece3.slice/crio-5c5e19604f278ad3eaab3c61f5374e639a7c4bb3aa0eed8c4a454420af5958f2 WatchSource:0}: Error finding container 5c5e19604f278ad3eaab3c61f5374e639a7c4bb3aa0eed8c4a454420af5958f2: Status 404 returned error can't find the container with id 5c5e19604f278ad3eaab3c61f5374e639a7c4bb3aa0eed8c4a454420af5958f2
Jul 15 21:19:22.223417 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:22.223387    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:19:22.223513 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:22.223465    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:19:22.223513 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:22.223495    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:19:22.986013 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:22.985964    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:19:23.081824 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:23.081778    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:19:23.088104 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:23.088070    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr" event={"ID":"4f22efb3-18ab-43a2-a936-4a717371ece3","Type":"ContainerStarted","Data":"18effc0af229947072eb9ff99e3516f9b27cc9c3c6218b7bd168b70571f4ffd1"}
Jul 15 21:19:23.088282 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:23.088164    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr" event={"ID":"4f22efb3-18ab-43a2-a936-4a717371ece3","Type":"ContainerStarted","Data":"5c5e19604f278ad3eaab3c61f5374e639a7c4bb3aa0eed8c4a454420af5958f2"}
Jul 15 21:19:23.123134 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:23.123073    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr" podStartSLOduration=2.12305225 podStartE2EDuration="2.12305225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:19:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:19:23.121724574 +0000 UTC m=+1896.270762631" watchObservedRunningTime="2026-07-15 21:19:23.12305225 +0000 UTC m=+1896.272090313"
Jul 15 21:19:23.165758 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:23.165710    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6e3236bd-86895f88cf-w589n"]
Jul 15 21:19:23.169396 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:23.169373    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6e3236bd-86895f88cf-w589n"
Jul 15 21:19:23.173682 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:23.173659    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 15 21:19:23.173682 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:23.173670    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-6e3236bd-serving-cert\""
Jul 15 21:19:23.173836 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:23.173709    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 15 21:19:23.174806 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:23.174789    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-s2l9d\""
Jul 15 21:19:23.183396 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:23.183364    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6e3236bd-86895f88cf-w589n"]
Jul 15 21:19:23.287649 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:23.287561    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:19:23.300815 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:23.300784    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5024762-4275-48ce-a7ff-43818cf49cf5-maas-api-tls\") pod \"maas-api-e2e-derive-6e3236bd-86895f88cf-w589n\" (UID: \"b5024762-4275-48ce-a7ff-43818cf49cf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6e3236bd-86895f88cf-w589n"
Jul 15 21:19:23.301026 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:23.300853    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5024762-4275-48ce-a7ff-43818cf49cf5-openshift-service-ca\") pod \"maas-api-e2e-derive-6e3236bd-86895f88cf-w589n\" (UID: \"b5024762-4275-48ce-a7ff-43818cf49cf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6e3236bd-86895f88cf-w589n"
Jul 15 21:19:23.301026 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:23.300957    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvct4\" (UniqueName: \"kubernetes.io/projected/b5024762-4275-48ce-a7ff-43818cf49cf5-kube-api-access-rvct4\") pod \"maas-api-e2e-derive-6e3236bd-86895f88cf-w589n\" (UID: \"b5024762-4275-48ce-a7ff-43818cf49cf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6e3236bd-86895f88cf-w589n"
Jul 15 21:19:23.401638 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:23.401597    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5024762-4275-48ce-a7ff-43818cf49cf5-maas-api-tls\") pod \"maas-api-e2e-derive-6e3236bd-86895f88cf-w589n\" (UID: \"b5024762-4275-48ce-a7ff-43818cf49cf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6e3236bd-86895f88cf-w589n"
Jul 15 21:19:23.401855 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:23.401655    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5024762-4275-48ce-a7ff-43818cf49cf5-openshift-service-ca\") pod \"maas-api-e2e-derive-6e3236bd-86895f88cf-w589n\" (UID: \"b5024762-4275-48ce-a7ff-43818cf49cf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6e3236bd-86895f88cf-w589n"
Jul 15 21:19:23.401855 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:23.401691    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvct4\" (UniqueName: \"kubernetes.io/projected/b5024762-4275-48ce-a7ff-43818cf49cf5-kube-api-access-rvct4\") pod \"maas-api-e2e-derive-6e3236bd-86895f88cf-w589n\" (UID: \"b5024762-4275-48ce-a7ff-43818cf49cf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6e3236bd-86895f88cf-w589n"
Jul 15 21:19:23.402470 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:23.402443    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5024762-4275-48ce-a7ff-43818cf49cf5-openshift-service-ca\") pod \"maas-api-e2e-derive-6e3236bd-86895f88cf-w589n\" (UID: \"b5024762-4275-48ce-a7ff-43818cf49cf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6e3236bd-86895f88cf-w589n"
Jul 15 21:19:23.404371 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:23.404349    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5024762-4275-48ce-a7ff-43818cf49cf5-maas-api-tls\") pod \"maas-api-e2e-derive-6e3236bd-86895f88cf-w589n\" (UID: \"b5024762-4275-48ce-a7ff-43818cf49cf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6e3236bd-86895f88cf-w589n"
Jul 15 21:19:23.412151 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:23.412123    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvct4\" (UniqueName: \"kubernetes.io/projected/b5024762-4275-48ce-a7ff-43818cf49cf5-kube-api-access-rvct4\") pod \"maas-api-e2e-derive-6e3236bd-86895f88cf-w589n\" (UID: \"b5024762-4275-48ce-a7ff-43818cf49cf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6e3236bd-86895f88cf-w589n"
Jul 15 21:19:23.429739 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:23.429710    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:19:23.482376 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:23.482343    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6e3236bd-86895f88cf-w589n"
Jul 15 21:19:23.619956 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:23.619925    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6e3236bd-86895f88cf-w589n"]
Jul 15 21:19:23.621775 ip-10-0-137-234 kubenswrapper[2566]: W0715 21:19:23.621754    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5024762_4275_48ce_a7ff_43818cf49cf5.slice/crio-00f56537984a6e6ae772ccad4b73ffc1d21fee7f03c7dca74af4ecbfc787b704 WatchSource:0}: Error finding container 00f56537984a6e6ae772ccad4b73ffc1d21fee7f03c7dca74af4ecbfc787b704: Status 404 returned error can't find the container with id 00f56537984a6e6ae772ccad4b73ffc1d21fee7f03c7dca74af4ecbfc787b704
Jul 15 21:19:23.851600 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:23.851511    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:19:23.951880 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:23.951845    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:19:24.072716 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:24.072683    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr"
Jul 15 21:19:24.078187 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:24.078158    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr"
Jul 15 21:19:24.094994 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:24.094960    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6e3236bd-86895f88cf-w589n" event={"ID":"b5024762-4275-48ce-a7ff-43818cf49cf5","Type":"ContainerStarted","Data":"00f56537984a6e6ae772ccad4b73ffc1d21fee7f03c7dca74af4ecbfc787b704"}
Jul 15 21:19:24.095189 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:24.095171    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr"
Jul 15 21:19:24.096157 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:24.096137    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr"
Jul 15 21:19:25.101737 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:25.101701    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6e3236bd-86895f88cf-w589n" event={"ID":"b5024762-4275-48ce-a7ff-43818cf49cf5","Type":"ContainerStarted","Data":"2e529a4dfc3dd50faeb070314a99c190109bf07b5eab596441dbeffba4e14cdf"}
Jul 15 21:19:25.131380 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:25.131330    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6e3236bd-86895f88cf-w589n" podStartSLOduration=1.623792172 podStartE2EDuration="2.131315355s" totalImagesPullingTime="507.523183ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:19:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:19:25.128422981 +0000 UTC m=+1898.277461037" watchObservedRunningTime="2026-07-15 21:19:25.131315355 +0000 UTC m=+1898.280353412"
Jul 15 21:19:26.107279 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:26.107249    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6e3236bd-86895f88cf-w589n"
Jul 15 21:19:26.675367 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:26.675342    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk"
Jul 15 21:19:26.834968 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:26.834933    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cc8ww\" (UniqueName: \"kubernetes.io/projected/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-kube-api-access-cc8ww\") pod \"8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4\" (UID: \"8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4\") "
Jul 15 21:19:26.835180 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:26.834988    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-workload-socket\") pod \"8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4\" (UID: \"8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4\") "
Jul 15 21:19:26.835180 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:26.835017    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-credential-socket\") pod \"8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4\" (UID: \"8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4\") "
Jul 15 21:19:26.835180 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:26.835039    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-istio-podinfo\") pod \"8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4\" (UID: \"8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4\") "
Jul 15 21:19:26.835180 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:26.835081    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-istiod-ca-cert\") pod \"8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4\" (UID: \"8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4\") "
Jul 15 21:19:26.835180 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:26.835119    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-istio-data\") pod \"8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4\" (UID: \"8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4\") "
Jul 15 21:19:26.835180 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:26.835167    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-workload-certs\") pod \"8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4\" (UID: \"8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4\") "
Jul 15 21:19:26.835509 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:26.835191    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-istio-envoy\") pod \"8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4\" (UID: \"8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4\") "
Jul 15 21:19:26.835509 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:26.835246    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-istio-token\") pod \"8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4\" (UID: \"8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4\") "
Jul 15 21:19:26.835509 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:26.835336    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4" (UID: "8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:19:26.835509 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:26.835370    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4" (UID: "8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:19:26.835702 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:26.835562    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-workload-socket\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:19:26.835702 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:26.835584    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-credential-socket\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:19:26.835702 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:26.835559    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4" (UID: "8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 21:19:26.835702 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:26.835657    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4" (UID: "8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:19:26.836353 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:26.836324    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-istio-data" (OuterVolumeSpecName: "istio-data") pod "8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4" (UID: "8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:19:26.837348 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:26.837308    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4" (UID: "8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 15 21:19:26.837443 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:26.837360    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-kube-api-access-cc8ww" (OuterVolumeSpecName: "kube-api-access-cc8ww") pod "8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4" (UID: "8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4"). InnerVolumeSpecName "kube-api-access-cc8ww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 21:19:26.837443 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:26.837390    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-istio-token" (OuterVolumeSpecName: "istio-token") pod "8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4" (UID: "8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 21:19:26.837443 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:26.837411    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4" (UID: "8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:19:26.936631 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:26.936595    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-istio-data\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:19:26.936631 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:26.936624    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-workload-certs\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:19:26.936631 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:26.936633    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-istio-envoy\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:19:26.936631 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:26.936641    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-istio-token\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:19:26.936864 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:26.936649    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc8ww\" (UniqueName: \"kubernetes.io/projected/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-kube-api-access-cc8ww\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:19:26.936864 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:26.936657    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-istio-podinfo\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:19:26.936864 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:26.936665    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4-istiod-ca-cert\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:19:27.113322 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:27.113226    2566 generic.go:356] "Generic (PLEG): container finished" podID="8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4" containerID="5c59798682e8ded9e13617de8622125bba3efdda8b52edd68e20112819a12fac" exitCode=0
Jul 15 21:19:27.113322 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:27.113277    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk"
Jul 15 21:19:27.113322 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:27.113307    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk" event={"ID":"8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4","Type":"ContainerDied","Data":"5c59798682e8ded9e13617de8622125bba3efdda8b52edd68e20112819a12fac"}
Jul 15 21:19:27.113836 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:27.113359    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk" event={"ID":"8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4","Type":"ContainerDied","Data":"a0227cfa6de59c532eea7471d5eba18807d2fe2445cf764dc78ce391780c5e7a"}
Jul 15 21:19:27.113836 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:27.113376    2566 scope.go:117] "RemoveContainer" containerID="5c59798682e8ded9e13617de8622125bba3efdda8b52edd68e20112819a12fac"
Jul 15 21:19:27.126563 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:27.126539    2566 scope.go:117] "RemoveContainer" containerID="0ee137a11412421adae02150dce1554c08a98ed6a83a914015e47a7b61f7f42b"
Jul 15 21:19:27.134367 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:27.134345    2566 scope.go:117] "RemoveContainer" containerID="5c59798682e8ded9e13617de8622125bba3efdda8b52edd68e20112819a12fac"
Jul 15 21:19:27.134631 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:19:27.134611    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c59798682e8ded9e13617de8622125bba3efdda8b52edd68e20112819a12fac\": container with ID starting with 5c59798682e8ded9e13617de8622125bba3efdda8b52edd68e20112819a12fac not found: ID does not exist" containerID="5c59798682e8ded9e13617de8622125bba3efdda8b52edd68e20112819a12fac"
Jul 15 21:19:27.134689 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:27.134640    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c59798682e8ded9e13617de8622125bba3efdda8b52edd68e20112819a12fac"} err="failed to get container status \"5c59798682e8ded9e13617de8622125bba3efdda8b52edd68e20112819a12fac\": rpc error: code = NotFound desc = could not find container \"5c59798682e8ded9e13617de8622125bba3efdda8b52edd68e20112819a12fac\": container with ID starting with 5c59798682e8ded9e13617de8622125bba3efdda8b52edd68e20112819a12fac not found: ID does not exist"
Jul 15 21:19:27.134689 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:27.134659    2566 scope.go:117] "RemoveContainer" containerID="0ee137a11412421adae02150dce1554c08a98ed6a83a914015e47a7b61f7f42b"
Jul 15 21:19:27.134881 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:19:27.134863    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ee137a11412421adae02150dce1554c08a98ed6a83a914015e47a7b61f7f42b\": container with ID starting with 0ee137a11412421adae02150dce1554c08a98ed6a83a914015e47a7b61f7f42b not found: ID does not exist" containerID="0ee137a11412421adae02150dce1554c08a98ed6a83a914015e47a7b61f7f42b"
Jul 15 21:19:27.134969 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:27.134906    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ee137a11412421adae02150dce1554c08a98ed6a83a914015e47a7b61f7f42b"} err="failed to get container status \"0ee137a11412421adae02150dce1554c08a98ed6a83a914015e47a7b61f7f42b\": rpc error: code = NotFound desc = could not find container \"0ee137a11412421adae02150dce1554c08a98ed6a83a914015e47a7b61f7f42b\": container with ID starting with 0ee137a11412421adae02150dce1554c08a98ed6a83a914015e47a7b61f7f42b not found: ID does not exist"
Jul 15 21:19:27.139534 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:27.139512    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk"]
Jul 15 21:19:27.143885 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:27.143864    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-af79a2b5-openshift-default-58c648b76f-p9kjk"]
Jul 15 21:19:27.319110 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:27.319070    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6e3236bd-w2kql"]
Jul 15 21:19:27.319524 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:27.319506    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4" containerName="istio-proxy"
Jul 15 21:19:27.319604 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:27.319527    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4" containerName="istio-proxy"
Jul 15 21:19:27.319677 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:27.319639    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4" containerName="istio-proxy"
Jul 15 21:19:27.319677 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:27.319651    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4" containerName="istio-proxy"
Jul 15 21:19:27.324379 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:27.324355    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6e3236bd-w2kql"
Jul 15 21:19:27.327460 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:27.327438    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zsrf5\""
Jul 15 21:19:27.331247 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:27.331223    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6e3236bd-w2kql"]
Jul 15 21:19:27.442098 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:27.442012    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc2b7d48-a365-4791-92a1-099a92df542f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6e3236bd-w2kql\" (UID: \"cc2b7d48-a365-4791-92a1-099a92df542f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6e3236bd-w2kql"
Jul 15 21:19:27.542949 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:27.542884    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc2b7d48-a365-4791-92a1-099a92df542f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6e3236bd-w2kql\" (UID: \"cc2b7d48-a365-4791-92a1-099a92df542f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6e3236bd-w2kql"
Jul 15 21:19:27.543662 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:27.543639    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc2b7d48-a365-4791-92a1-099a92df542f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6e3236bd-w2kql\" (UID: \"cc2b7d48-a365-4791-92a1-099a92df542f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6e3236bd-w2kql"
Jul 15 21:19:27.636199 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:27.636161    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6e3236bd-w2kql"
Jul 15 21:19:27.757853 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:27.757812    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4" path="/var/lib/kubelet/pods/8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4/volumes"
Jul 15 21:19:27.772164 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:27.772142    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6e3236bd-w2kql"]
Jul 15 21:19:27.774308 ip-10-0-137-234 kubenswrapper[2566]: W0715 21:19:27.774276    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc2b7d48_a365_4791_92a1_099a92df542f.slice/crio-7d2922e4061ce29523441414853e894cbad5a4d1b458991ec5a6da7d06a48dc6 WatchSource:0}: Error finding container 7d2922e4061ce29523441414853e894cbad5a4d1b458991ec5a6da7d06a48dc6: Status 404 returned error can't find the container with id 7d2922e4061ce29523441414853e894cbad5a4d1b458991ec5a6da7d06a48dc6
Jul 15 21:19:28.120856 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:28.120821    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6e3236bd-w2kql" event={"ID":"cc2b7d48-a365-4791-92a1-099a92df542f","Type":"ContainerStarted","Data":"87f0f0ac566e8f46317ad58bc985013d77a0673e8ed53b4eb46585fd5269f7b0"}
Jul 15 21:19:28.121270 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:28.120873    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6e3236bd-w2kql" event={"ID":"cc2b7d48-a365-4791-92a1-099a92df542f","Type":"ContainerStarted","Data":"7d2922e4061ce29523441414853e894cbad5a4d1b458991ec5a6da7d06a48dc6"}
Jul 15 21:19:28.139600 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:28.139549    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6e3236bd-w2kql" podStartSLOduration=1.139532967 podStartE2EDuration="1.139532967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:19:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:19:28.137312266 +0000 UTC m=+1901.286350323" watchObservedRunningTime="2026-07-15 21:19:28.139532967 +0000 UTC m=+1901.288571023"
Jul 15 21:19:29.127469 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:29.127431    2566 generic.go:356] "Generic (PLEG): container finished" podID="cc2b7d48-a365-4791-92a1-099a92df542f" containerID="87f0f0ac566e8f46317ad58bc985013d77a0673e8ed53b4eb46585fd5269f7b0" exitCode=7
Jul 15 21:19:29.127914 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:29.127514    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6e3236bd-w2kql" event={"ID":"cc2b7d48-a365-4791-92a1-099a92df542f","Type":"ContainerDied","Data":"87f0f0ac566e8f46317ad58bc985013d77a0673e8ed53b4eb46585fd5269f7b0"}
Jul 15 21:19:29.127914 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:29.127689    2566 scope.go:117] "RemoveContainer" containerID="87f0f0ac566e8f46317ad58bc985013d77a0673e8ed53b4eb46585fd5269f7b0"
Jul 15 21:19:30.134202 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:30.134103    2566 generic.go:356] "Generic (PLEG): container finished" podID="cc2b7d48-a365-4791-92a1-099a92df542f" containerID="55a613346ff5f9c089a419b595bab8f7fb8f5c6794aac1aab80d7c219951a7c4" exitCode=7
Jul 15 21:19:30.134202 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:30.134185    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6e3236bd-w2kql" event={"ID":"cc2b7d48-a365-4791-92a1-099a92df542f","Type":"ContainerDied","Data":"55a613346ff5f9c089a419b595bab8f7fb8f5c6794aac1aab80d7c219951a7c4"}
Jul 15 21:19:30.134729 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:30.134241    2566 scope.go:117] "RemoveContainer" containerID="87f0f0ac566e8f46317ad58bc985013d77a0673e8ed53b4eb46585fd5269f7b0"
Jul 15 21:19:30.134729 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:30.134521    2566 scope.go:117] "RemoveContainer" containerID="55a613346ff5f9c089a419b595bab8f7fb8f5c6794aac1aab80d7c219951a7c4"
Jul 15 21:19:30.134729 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:19:30.134714    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6e3236bd-w2kql_odh-ai-gateway-infra(cc2b7d48-a365-4791-92a1-099a92df542f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6e3236bd-w2kql" podUID="cc2b7d48-a365-4791-92a1-099a92df542f"
Jul 15 21:19:31.141117 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:31.141085    2566 scope.go:117] "RemoveContainer" containerID="55a613346ff5f9c089a419b595bab8f7fb8f5c6794aac1aab80d7c219951a7c4"
Jul 15 21:19:31.141560 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:19:31.141355    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6e3236bd-w2kql_odh-ai-gateway-infra(cc2b7d48-a365-4791-92a1-099a92df542f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6e3236bd-w2kql" podUID="cc2b7d48-a365-4791-92a1-099a92df542f"
Jul 15 21:19:32.119562 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:32.119524    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6e3236bd-86895f88cf-w589n"
Jul 15 21:19:34.469539 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:34.469501    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:19:34.533415 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:34.533369    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:19:37.176118 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:37.176079    2566 generic.go:356] "Generic (PLEG): container finished" podID="4f22efb3-18ab-43a2-a936-4a717371ece3" containerID="18effc0af229947072eb9ff99e3516f9b27cc9c3c6218b7bd168b70571f4ffd1" exitCode=0
Jul 15 21:19:37.176563 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:37.176159    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr" event={"ID":"4f22efb3-18ab-43a2-a936-4a717371ece3","Type":"ContainerDied","Data":"18effc0af229947072eb9ff99e3516f9b27cc9c3c6218b7bd168b70571f4ffd1"}
Jul 15 21:19:37.176627 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:37.176610    2566 scope.go:117] "RemoveContainer" containerID="18effc0af229947072eb9ff99e3516f9b27cc9c3c6218b7bd168b70571f4ffd1"
Jul 15 21:19:37.177714 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:37.177686    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:19:37.177798 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:37.177779    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:19:37.177847 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:37.177819    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:19:37.784664 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:37.784621    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:19:38.192288 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:38.192181    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr" event={"ID":"4f22efb3-18ab-43a2-a936-4a717371ece3","Type":"ContainerStarted","Data":"e14aed7a7e25f933b6e663d2bbeff0ff68d0d000995e47e83c1cbe81d1c49708"}
Jul 15 21:19:39.072959 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:39.072921    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr"
Jul 15 21:19:39.074038 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:39.074015    2566 patch_prober.go:28] interesting pod/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Jul 15 21:19:39.074105 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:39.074077    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr" podUID="4f22efb3-18ab-43a2-a936-4a717371ece3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Jul 15 21:19:39.095702 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:39.095672    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr"
Jul 15 21:19:40.073350 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:40.073308    2566 patch_prober.go:28] interesting pod/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Jul 15 21:19:40.073822 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:40.073380    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr" podUID="4f22efb3-18ab-43a2-a936-4a717371ece3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Jul 15 21:19:41.073092 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:41.073050    2566 patch_prober.go:28] interesting pod/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Jul 15 21:19:41.073274 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:41.073115    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr" podUID="4f22efb3-18ab-43a2-a936-4a717371ece3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Jul 15 21:19:42.077651 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:42.077621    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr"
Jul 15 21:19:42.078722 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:42.078699    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr"
Jul 15 21:19:44.751150 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:44.751116    2566 scope.go:117] "RemoveContainer" containerID="55a613346ff5f9c089a419b595bab8f7fb8f5c6794aac1aab80d7c219951a7c4"
Jul 15 21:19:45.235583 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:45.235550    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6e3236bd-w2kql" event={"ID":"cc2b7d48-a365-4791-92a1-099a92df542f","Type":"ContainerStarted","Data":"0d5c83eb57852f2d77d3e1809f4d671819ec61bf7a955f5e03fd953ade4e95d8"}
Jul 15 21:19:46.241941 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:46.241907    2566 generic.go:356] "Generic (PLEG): container finished" podID="cc2b7d48-a365-4791-92a1-099a92df542f" containerID="0d5c83eb57852f2d77d3e1809f4d671819ec61bf7a955f5e03fd953ade4e95d8" exitCode=0
Jul 15 21:19:46.241941 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:46.241922    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6e3236bd-w2kql" event={"ID":"cc2b7d48-a365-4791-92a1-099a92df542f","Type":"ContainerDied","Data":"0d5c83eb57852f2d77d3e1809f4d671819ec61bf7a955f5e03fd953ade4e95d8"}
Jul 15 21:19:46.242442 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:46.241981    2566 scope.go:117] "RemoveContainer" containerID="55a613346ff5f9c089a419b595bab8f7fb8f5c6794aac1aab80d7c219951a7c4"
Jul 15 21:19:46.269634 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:46.269606    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6e3236bd-w2kql"]
Jul 15 21:19:47.375183 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:47.375161    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6e3236bd-w2kql"
Jul 15 21:19:47.436096 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:47.436067    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc2b7d48-a365-4791-92a1-099a92df542f-maas-api-service-ca\") pod \"cc2b7d48-a365-4791-92a1-099a92df542f\" (UID: \"cc2b7d48-a365-4791-92a1-099a92df542f\") "
Jul 15 21:19:47.436419 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:47.436396    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc2b7d48-a365-4791-92a1-099a92df542f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cc2b7d48-a365-4791-92a1-099a92df542f" (UID: "cc2b7d48-a365-4791-92a1-099a92df542f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 21:19:47.537133 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:47.537049    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc2b7d48-a365-4791-92a1-099a92df542f-maas-api-service-ca\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:19:48.254567 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:48.254528    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6e3236bd-w2kql" event={"ID":"cc2b7d48-a365-4791-92a1-099a92df542f","Type":"ContainerDied","Data":"7d2922e4061ce29523441414853e894cbad5a4d1b458991ec5a6da7d06a48dc6"}
Jul 15 21:19:48.254772 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:48.254588    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d2922e4061ce29523441414853e894cbad5a4d1b458991ec5a6da7d06a48dc6"
Jul 15 21:19:48.254772 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:48.254543    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6e3236bd-w2kql"
Jul 15 21:19:48.288075 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:48.288038    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6e3236bd-w2kql"]
Jul 15 21:19:48.293976 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:48.293949    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6e3236bd-w2kql"]
Jul 15 21:19:49.756516 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:19:49.756470    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc2b7d48-a365-4791-92a1-099a92df542f" path="/var/lib/kubelet/pods/cc2b7d48-a365-4791-92a1-099a92df542f/volumes"
Jul 15 21:20:27.373021 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:27.368169    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6e3236bd-t7qlb"]
Jul 15 21:20:27.373021 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:27.368765    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc2b7d48-a365-4791-92a1-099a92df542f" containerName="revoke-keys"
Jul 15 21:20:27.373021 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:27.368783    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc2b7d48-a365-4791-92a1-099a92df542f" containerName="revoke-keys"
Jul 15 21:20:27.373021 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:27.368814    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4" containerName="istio-proxy"
Jul 15 21:20:27.373021 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:27.368820    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f3c091d-d2cb-43aa-bd7b-a25777c8a2c4" containerName="istio-proxy"
Jul 15 21:20:27.373021 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:27.368837    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc2b7d48-a365-4791-92a1-099a92df542f" containerName="revoke-keys"
Jul 15 21:20:27.373021 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:27.368842    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc2b7d48-a365-4791-92a1-099a92df542f" containerName="revoke-keys"
Jul 15 21:20:27.373021 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:27.368959    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc2b7d48-a365-4791-92a1-099a92df542f" containerName="revoke-keys"
Jul 15 21:20:27.373021 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:27.368970    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc2b7d48-a365-4791-92a1-099a92df542f" containerName="revoke-keys"
Jul 15 21:20:27.373021 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:27.372805    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6e3236bd-t7qlb"
Jul 15 21:20:27.376288 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:27.376187    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zsrf5\""
Jul 15 21:20:27.379030 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:27.379002    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6e3236bd-t7qlb"]
Jul 15 21:20:27.380143 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:27.380110    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb1054f6-f436-4f1b-a556-a6d4c394a44f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6e3236bd-t7qlb\" (UID: \"eb1054f6-f436-4f1b-a556-a6d4c394a44f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6e3236bd-t7qlb"
Jul 15 21:20:27.480772 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:27.480736    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb1054f6-f436-4f1b-a556-a6d4c394a44f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6e3236bd-t7qlb\" (UID: \"eb1054f6-f436-4f1b-a556-a6d4c394a44f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6e3236bd-t7qlb"
Jul 15 21:20:27.481424 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:27.481397    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb1054f6-f436-4f1b-a556-a6d4c394a44f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6e3236bd-t7qlb\" (UID: \"eb1054f6-f436-4f1b-a556-a6d4c394a44f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6e3236bd-t7qlb"
Jul 15 21:20:27.685417 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:27.685329    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6e3236bd-t7qlb"
Jul 15 21:20:27.827869 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:27.827790    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6e3236bd-t7qlb"]
Jul 15 21:20:27.830258 ip-10-0-137-234 kubenswrapper[2566]: W0715 21:20:27.830232    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb1054f6_f436_4f1b_a556_a6d4c394a44f.slice/crio-076f3b9b612958fb518955d5ed85ec46c4c0d2a0ca5e8e413ad68f62f78edff4 WatchSource:0}: Error finding container 076f3b9b612958fb518955d5ed85ec46c4c0d2a0ca5e8e413ad68f62f78edff4: Status 404 returned error can't find the container with id 076f3b9b612958fb518955d5ed85ec46c4c0d2a0ca5e8e413ad68f62f78edff4
Jul 15 21:20:28.450362 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:28.450320    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6e3236bd-t7qlb" event={"ID":"eb1054f6-f436-4f1b-a556-a6d4c394a44f","Type":"ContainerStarted","Data":"d9905d8be085d2c530fecd4a1dc7a1cc08e208752b8791f3ca87cc2bb51ee8b2"}
Jul 15 21:20:28.450848 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:28.450388    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6e3236bd-t7qlb" event={"ID":"eb1054f6-f436-4f1b-a556-a6d4c394a44f","Type":"ContainerStarted","Data":"076f3b9b612958fb518955d5ed85ec46c4c0d2a0ca5e8e413ad68f62f78edff4"}
Jul 15 21:20:28.468603 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:28.468553    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6e3236bd-t7qlb" podStartSLOduration=1.468533031 podStartE2EDuration="1.468533031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:20:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:20:28.467969967 +0000 UTC m=+1961.617008024" watchObservedRunningTime="2026-07-15 21:20:28.468533031 +0000 UTC m=+1961.617571089"
Jul 15 21:20:29.457596 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:29.457560    2566 generic.go:356] "Generic (PLEG): container finished" podID="eb1054f6-f436-4f1b-a556-a6d4c394a44f" containerID="d9905d8be085d2c530fecd4a1dc7a1cc08e208752b8791f3ca87cc2bb51ee8b2" exitCode=0
Jul 15 21:20:29.458033 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:29.457668    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6e3236bd-t7qlb" event={"ID":"eb1054f6-f436-4f1b-a556-a6d4c394a44f","Type":"ContainerDied","Data":"d9905d8be085d2c530fecd4a1dc7a1cc08e208752b8791f3ca87cc2bb51ee8b2"}
Jul 15 21:20:30.597269 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:30.597246    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6e3236bd-t7qlb"
Jul 15 21:20:30.602985 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:30.602965    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb1054f6-f436-4f1b-a556-a6d4c394a44f-maas-api-service-ca\") pod \"eb1054f6-f436-4f1b-a556-a6d4c394a44f\" (UID: \"eb1054f6-f436-4f1b-a556-a6d4c394a44f\") "
Jul 15 21:20:30.603317 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:30.603298    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb1054f6-f436-4f1b-a556-a6d4c394a44f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eb1054f6-f436-4f1b-a556-a6d4c394a44f" (UID: "eb1054f6-f436-4f1b-a556-a6d4c394a44f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 21:20:30.703985 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:30.703947    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb1054f6-f436-4f1b-a556-a6d4c394a44f-maas-api-service-ca\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:20:31.468620 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:31.468590    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6e3236bd-t7qlb"
Jul 15 21:20:31.468620 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:31.468599    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6e3236bd-t7qlb" event={"ID":"eb1054f6-f436-4f1b-a556-a6d4c394a44f","Type":"ContainerDied","Data":"076f3b9b612958fb518955d5ed85ec46c4c0d2a0ca5e8e413ad68f62f78edff4"}
Jul 15 21:20:31.468824 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:31.468643    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="076f3b9b612958fb518955d5ed85ec46c4c0d2a0ca5e8e413ad68f62f78edff4"
Jul 15 21:20:37.973184 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:37.973152    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6e3236bd-86895f88cf-w589n"]
Jul 15 21:20:37.973553 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:37.973343    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6e3236bd-86895f88cf-w589n" podUID="b5024762-4275-48ce-a7ff-43818cf49cf5" containerName="maas-api" containerID="cri-o://2e529a4dfc3dd50faeb070314a99c190109bf07b5eab596441dbeffba4e14cdf" gracePeriod=30
Jul 15 21:20:38.003122 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:38.003090    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:20:38.224026 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:38.223962    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6e3236bd-86895f88cf-w589n"
Jul 15 21:20:38.256311 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:38.256273    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5024762-4275-48ce-a7ff-43818cf49cf5-maas-api-tls\") pod \"b5024762-4275-48ce-a7ff-43818cf49cf5\" (UID: \"b5024762-4275-48ce-a7ff-43818cf49cf5\") "
Jul 15 21:20:38.256464 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:38.256319    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5024762-4275-48ce-a7ff-43818cf49cf5-openshift-service-ca\") pod \"b5024762-4275-48ce-a7ff-43818cf49cf5\" (UID: \"b5024762-4275-48ce-a7ff-43818cf49cf5\") "
Jul 15 21:20:38.256464 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:38.256353    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvct4\" (UniqueName: \"kubernetes.io/projected/b5024762-4275-48ce-a7ff-43818cf49cf5-kube-api-access-rvct4\") pod \"b5024762-4275-48ce-a7ff-43818cf49cf5\" (UID: \"b5024762-4275-48ce-a7ff-43818cf49cf5\") "
Jul 15 21:20:38.256688 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:38.256662    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5024762-4275-48ce-a7ff-43818cf49cf5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b5024762-4275-48ce-a7ff-43818cf49cf5" (UID: "b5024762-4275-48ce-a7ff-43818cf49cf5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 21:20:38.258405 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:38.258381    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5024762-4275-48ce-a7ff-43818cf49cf5-kube-api-access-rvct4" (OuterVolumeSpecName: "kube-api-access-rvct4") pod "b5024762-4275-48ce-a7ff-43818cf49cf5" (UID: "b5024762-4275-48ce-a7ff-43818cf49cf5"). InnerVolumeSpecName "kube-api-access-rvct4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 21:20:38.258506 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:38.258479    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5024762-4275-48ce-a7ff-43818cf49cf5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b5024762-4275-48ce-a7ff-43818cf49cf5" (UID: "b5024762-4275-48ce-a7ff-43818cf49cf5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 21:20:38.357747 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:38.357707    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvct4\" (UniqueName: \"kubernetes.io/projected/b5024762-4275-48ce-a7ff-43818cf49cf5-kube-api-access-rvct4\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:20:38.357747 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:38.357740    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5024762-4275-48ce-a7ff-43818cf49cf5-maas-api-tls\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:20:38.357747 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:38.357750    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5024762-4275-48ce-a7ff-43818cf49cf5-openshift-service-ca\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:20:38.505103 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:38.505013    2566 generic.go:356] "Generic (PLEG): container finished" podID="b5024762-4275-48ce-a7ff-43818cf49cf5" containerID="2e529a4dfc3dd50faeb070314a99c190109bf07b5eab596441dbeffba4e14cdf" exitCode=0
Jul 15 21:20:38.505103 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:38.505074    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6e3236bd-86895f88cf-w589n"
Jul 15 21:20:38.505273 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:38.505103    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6e3236bd-86895f88cf-w589n" event={"ID":"b5024762-4275-48ce-a7ff-43818cf49cf5","Type":"ContainerDied","Data":"2e529a4dfc3dd50faeb070314a99c190109bf07b5eab596441dbeffba4e14cdf"}
Jul 15 21:20:38.505273 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:38.505182    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6e3236bd-86895f88cf-w589n" event={"ID":"b5024762-4275-48ce-a7ff-43818cf49cf5","Type":"ContainerDied","Data":"00f56537984a6e6ae772ccad4b73ffc1d21fee7f03c7dca74af4ecbfc787b704"}
Jul 15 21:20:38.505273 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:38.505204    2566 scope.go:117] "RemoveContainer" containerID="2e529a4dfc3dd50faeb070314a99c190109bf07b5eab596441dbeffba4e14cdf"
Jul 15 21:20:38.514143 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:38.514121    2566 scope.go:117] "RemoveContainer" containerID="2e529a4dfc3dd50faeb070314a99c190109bf07b5eab596441dbeffba4e14cdf"
Jul 15 21:20:38.514385 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:20:38.514364    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e529a4dfc3dd50faeb070314a99c190109bf07b5eab596441dbeffba4e14cdf\": container with ID starting with 2e529a4dfc3dd50faeb070314a99c190109bf07b5eab596441dbeffba4e14cdf not found: ID does not exist" containerID="2e529a4dfc3dd50faeb070314a99c190109bf07b5eab596441dbeffba4e14cdf"
Jul 15 21:20:38.514440 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:38.514393    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e529a4dfc3dd50faeb070314a99c190109bf07b5eab596441dbeffba4e14cdf"} err="failed to get container status \"2e529a4dfc3dd50faeb070314a99c190109bf07b5eab596441dbeffba4e14cdf\": rpc error: code = NotFound desc = could not find container \"2e529a4dfc3dd50faeb070314a99c190109bf07b5eab596441dbeffba4e14cdf\": container with ID starting with 2e529a4dfc3dd50faeb070314a99c190109bf07b5eab596441dbeffba4e14cdf not found: ID does not exist"
Jul 15 21:20:38.533653 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:38.533625    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6e3236bd-86895f88cf-w589n"]
Jul 15 21:20:38.538069 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:38.538049    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6e3236bd-86895f88cf-w589n"]
Jul 15 21:20:39.757195 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:39.757164    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5024762-4275-48ce-a7ff-43818cf49cf5" path="/var/lib/kubelet/pods/b5024762-4275-48ce-a7ff-43818cf49cf5/volumes"
Jul 15 21:20:42.461823 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:42.461790    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:20:48.638424 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:48.638377    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:20:48.704687 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:48.704632    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:20:52.670253 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:52.670215    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr"]
Jul 15 21:20:52.670716 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:52.670480    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr" podUID="4f22efb3-18ab-43a2-a936-4a717371ece3" containerName="istio-proxy" containerID="cri-o://e14aed7a7e25f933b6e663d2bbeff0ff68d0d000995e47e83c1cbe81d1c49708" gracePeriod=30
Jul 15 21:20:52.720391 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:52.720326    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:20:53.353489 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:53.353448    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:20:53.436418 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:53.436386    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:20:53.536947 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:53.536887    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:20:55.086592 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:55.086544    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:20:57.080561 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:57.080526    2566 patch_prober.go:28] interesting pod/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 15 21:20:57.080963 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:57.080585    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr" podUID="4f22efb3-18ab-43a2-a936-4a717371ece3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 15 21:20:57.920744 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:57.920716    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr"
Jul 15 21:20:58.033844 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:58.033809    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f22efb3-18ab-43a2-a936-4a717371ece3-workload-certs\") pod \"4f22efb3-18ab-43a2-a936-4a717371ece3\" (UID: \"4f22efb3-18ab-43a2-a936-4a717371ece3\") "
Jul 15 21:20:58.034029 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:58.033857    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f22efb3-18ab-43a2-a936-4a717371ece3-credential-socket\") pod \"4f22efb3-18ab-43a2-a936-4a717371ece3\" (UID: \"4f22efb3-18ab-43a2-a936-4a717371ece3\") "
Jul 15 21:20:58.034029 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:58.033883    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f22efb3-18ab-43a2-a936-4a717371ece3-istio-envoy\") pod \"4f22efb3-18ab-43a2-a936-4a717371ece3\" (UID: \"4f22efb3-18ab-43a2-a936-4a717371ece3\") "
Jul 15 21:20:58.034029 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:58.033931    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f22efb3-18ab-43a2-a936-4a717371ece3-istio-data\") pod \"4f22efb3-18ab-43a2-a936-4a717371ece3\" (UID: \"4f22efb3-18ab-43a2-a936-4a717371ece3\") "
Jul 15 21:20:58.034029 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:58.033960    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f22efb3-18ab-43a2-a936-4a717371ece3-istiod-ca-cert\") pod \"4f22efb3-18ab-43a2-a936-4a717371ece3\" (UID: \"4f22efb3-18ab-43a2-a936-4a717371ece3\") "
Jul 15 21:20:58.034029 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:58.033998    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f22efb3-18ab-43a2-a936-4a717371ece3-istio-token\") pod \"4f22efb3-18ab-43a2-a936-4a717371ece3\" (UID: \"4f22efb3-18ab-43a2-a936-4a717371ece3\") "
Jul 15 21:20:58.034029 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:58.034021    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9cdj\" (UniqueName: \"kubernetes.io/projected/4f22efb3-18ab-43a2-a936-4a717371ece3-kube-api-access-c9cdj\") pod \"4f22efb3-18ab-43a2-a936-4a717371ece3\" (UID: \"4f22efb3-18ab-43a2-a936-4a717371ece3\") "
Jul 15 21:20:58.034295 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:58.034057    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f22efb3-18ab-43a2-a936-4a717371ece3-istio-podinfo\") pod \"4f22efb3-18ab-43a2-a936-4a717371ece3\" (UID: \"4f22efb3-18ab-43a2-a936-4a717371ece3\") "
Jul 15 21:20:58.034295 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:58.034148    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f22efb3-18ab-43a2-a936-4a717371ece3-workload-socket\") pod \"4f22efb3-18ab-43a2-a936-4a717371ece3\" (UID: \"4f22efb3-18ab-43a2-a936-4a717371ece3\") "
Jul 15 21:20:58.034295 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:58.034157    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f22efb3-18ab-43a2-a936-4a717371ece3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4f22efb3-18ab-43a2-a936-4a717371ece3" (UID: "4f22efb3-18ab-43a2-a936-4a717371ece3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:20:58.034295 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:58.034251    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f22efb3-18ab-43a2-a936-4a717371ece3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4f22efb3-18ab-43a2-a936-4a717371ece3" (UID: "4f22efb3-18ab-43a2-a936-4a717371ece3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:20:58.034515 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:58.034438    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f22efb3-18ab-43a2-a936-4a717371ece3-credential-socket\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:20:58.034515 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:58.034459    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f22efb3-18ab-43a2-a936-4a717371ece3-workload-certs\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:20:58.034515 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:58.034460    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f22efb3-18ab-43a2-a936-4a717371ece3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4f22efb3-18ab-43a2-a936-4a717371ece3" (UID: "4f22efb3-18ab-43a2-a936-4a717371ece3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:20:58.034639 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:58.034531    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f22efb3-18ab-43a2-a936-4a717371ece3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4f22efb3-18ab-43a2-a936-4a717371ece3" (UID: "4f22efb3-18ab-43a2-a936-4a717371ece3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 21:20:58.035117 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:58.035091    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f22efb3-18ab-43a2-a936-4a717371ece3-istio-data" (OuterVolumeSpecName: "istio-data") pod "4f22efb3-18ab-43a2-a936-4a717371ece3" (UID: "4f22efb3-18ab-43a2-a936-4a717371ece3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:20:58.036240 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:58.036215    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f22efb3-18ab-43a2-a936-4a717371ece3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4f22efb3-18ab-43a2-a936-4a717371ece3" (UID: "4f22efb3-18ab-43a2-a936-4a717371ece3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 15 21:20:58.036338 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:58.036266    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f22efb3-18ab-43a2-a936-4a717371ece3-kube-api-access-c9cdj" (OuterVolumeSpecName: "kube-api-access-c9cdj") pod "4f22efb3-18ab-43a2-a936-4a717371ece3" (UID: "4f22efb3-18ab-43a2-a936-4a717371ece3"). InnerVolumeSpecName "kube-api-access-c9cdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 21:20:58.036591 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:58.036563    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f22efb3-18ab-43a2-a936-4a717371ece3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4f22efb3-18ab-43a2-a936-4a717371ece3" (UID: "4f22efb3-18ab-43a2-a936-4a717371ece3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:20:58.036591 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:58.036569    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f22efb3-18ab-43a2-a936-4a717371ece3-istio-token" (OuterVolumeSpecName: "istio-token") pod "4f22efb3-18ab-43a2-a936-4a717371ece3" (UID: "4f22efb3-18ab-43a2-a936-4a717371ece3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 21:20:58.135688 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:58.135651    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f22efb3-18ab-43a2-a936-4a717371ece3-istio-envoy\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:20:58.135688 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:58.135681    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f22efb3-18ab-43a2-a936-4a717371ece3-istio-data\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:20:58.135688 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:58.135690    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f22efb3-18ab-43a2-a936-4a717371ece3-istiod-ca-cert\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:20:58.136152 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:58.135700    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f22efb3-18ab-43a2-a936-4a717371ece3-istio-token\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:20:58.136152 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:58.135710    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9cdj\" (UniqueName: \"kubernetes.io/projected/4f22efb3-18ab-43a2-a936-4a717371ece3-kube-api-access-c9cdj\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:20:58.136152 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:58.135720    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f22efb3-18ab-43a2-a936-4a717371ece3-istio-podinfo\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:20:58.136152 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:58.135729    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f22efb3-18ab-43a2-a936-4a717371ece3-workload-socket\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:20:58.606680 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:58.606639    2566 generic.go:356] "Generic (PLEG): container finished" podID="4f22efb3-18ab-43a2-a936-4a717371ece3" containerID="e14aed7a7e25f933b6e663d2bbeff0ff68d0d000995e47e83c1cbe81d1c49708" exitCode=0
Jul 15 21:20:58.606680 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:58.606683    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr"
Jul 15 21:20:58.606913 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:58.606676    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr" event={"ID":"4f22efb3-18ab-43a2-a936-4a717371ece3","Type":"ContainerDied","Data":"e14aed7a7e25f933b6e663d2bbeff0ff68d0d000995e47e83c1cbe81d1c49708"}
Jul 15 21:20:58.606913 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:58.606813    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr" event={"ID":"4f22efb3-18ab-43a2-a936-4a717371ece3","Type":"ContainerDied","Data":"5c5e19604f278ad3eaab3c61f5374e639a7c4bb3aa0eed8c4a454420af5958f2"}
Jul 15 21:20:58.606913 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:58.606852    2566 scope.go:117] "RemoveContainer" containerID="e14aed7a7e25f933b6e663d2bbeff0ff68d0d000995e47e83c1cbe81d1c49708"
Jul 15 21:20:58.615808 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:58.615786    2566 scope.go:117] "RemoveContainer" containerID="18effc0af229947072eb9ff99e3516f9b27cc9c3c6218b7bd168b70571f4ffd1"
Jul 15 21:20:58.624117 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:58.624098    2566 scope.go:117] "RemoveContainer" containerID="e14aed7a7e25f933b6e663d2bbeff0ff68d0d000995e47e83c1cbe81d1c49708"
Jul 15 21:20:58.624402 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:20:58.624376    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e14aed7a7e25f933b6e663d2bbeff0ff68d0d000995e47e83c1cbe81d1c49708\": container with ID starting with e14aed7a7e25f933b6e663d2bbeff0ff68d0d000995e47e83c1cbe81d1c49708 not found: ID does not exist" containerID="e14aed7a7e25f933b6e663d2bbeff0ff68d0d000995e47e83c1cbe81d1c49708"
Jul 15 21:20:58.624448 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:58.624411    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e14aed7a7e25f933b6e663d2bbeff0ff68d0d000995e47e83c1cbe81d1c49708"} err="failed to get container status \"e14aed7a7e25f933b6e663d2bbeff0ff68d0d000995e47e83c1cbe81d1c49708\": rpc error: code = NotFound desc = could not find container \"e14aed7a7e25f933b6e663d2bbeff0ff68d0d000995e47e83c1cbe81d1c49708\": container with ID starting with e14aed7a7e25f933b6e663d2bbeff0ff68d0d000995e47e83c1cbe81d1c49708 not found: ID does not exist"
Jul 15 21:20:58.624448 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:58.624428    2566 scope.go:117] "RemoveContainer" containerID="18effc0af229947072eb9ff99e3516f9b27cc9c3c6218b7bd168b70571f4ffd1"
Jul 15 21:20:58.624655 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:20:58.624639    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18effc0af229947072eb9ff99e3516f9b27cc9c3c6218b7bd168b70571f4ffd1\": container with ID starting with 18effc0af229947072eb9ff99e3516f9b27cc9c3c6218b7bd168b70571f4ffd1 not found: ID does not exist" containerID="18effc0af229947072eb9ff99e3516f9b27cc9c3c6218b7bd168b70571f4ffd1"
Jul 15 21:20:58.624697 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:58.624659    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18effc0af229947072eb9ff99e3516f9b27cc9c3c6218b7bd168b70571f4ffd1"} err="failed to get container status \"18effc0af229947072eb9ff99e3516f9b27cc9c3c6218b7bd168b70571f4ffd1\": rpc error: code = NotFound desc = could not find container \"18effc0af229947072eb9ff99e3516f9b27cc9c3c6218b7bd168b70571f4ffd1\": container with ID starting with 18effc0af229947072eb9ff99e3516f9b27cc9c3c6218b7bd168b70571f4ffd1 not found: ID does not exist"
Jul 15 21:20:58.631821 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:58.631800    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr"]
Jul 15 21:20:58.638683 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:58.638659    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-6e3236bd-openshift-default-69d54b79c5-xkmxr"]
Jul 15 21:20:59.251710 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:59.251667    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:20:59.757176 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:20:59.757142    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f22efb3-18ab-43a2-a936-4a717371ece3" path="/var/lib/kubelet/pods/4f22efb3-18ab-43a2-a936-4a717371ece3/volumes"
Jul 15 21:21:01.467269 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:21:01.467231    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:21:01.572232 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:21:01.572202    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:21:01.670512 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:21:01.670465    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:21:01.767849 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:21:01.767758    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:21:02.239374 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:21:02.239335    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:21:28.267670 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:21:28.267619    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:21:34.550774 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:21:34.550732    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:21:34.638189 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:21:34.638143    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:21:34.737493 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:21:34.737455    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:21:40.225808 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:21:40.225770    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:21:40.297078 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:21:40.297037    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:21:42.580953 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:21:42.580915    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:21:47.971701 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:21:47.971668    2566 scope.go:117] "RemoveContainer" containerID="96b9db41f52f295275ba34df02df72eea04e3559342973fd554699b0e3e41fcb"
Jul 15 21:21:50.753161 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:21:50.753125    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:21:57.399183 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:21:57.399145    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:21:57.503735 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:21:57.503692    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:21:57.608097 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:21:57.608061    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:22:08.232391 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:08.232312    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:22:09.113821 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:09.113780    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:22:15.452851 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:15.452810    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:22:15.553800 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:15.553766    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:22:15.656623 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:15.656586    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:22:15.772112 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:15.772026    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:22:15.879440 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:15.879409    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:22:16.681429 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:16.681393    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:22:16.778339 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:16.778305    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:22:16.877458 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:16.877426    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:22:18.701613 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:18.701581    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:22:24.983092 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:24.983055    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:22:25.055502 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:25.055467    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:22:25.731449 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:25.731409    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:22:32.181313 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:32.181272    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:22:32.266506 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:32.266473    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:22:32.488117 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:32.488082    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:22:32.789148 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:32.789069    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt"]
Jul 15 21:22:32.789407 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:32.789395    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f22efb3-18ab-43a2-a936-4a717371ece3" containerName="istio-proxy"
Jul 15 21:22:32.789455 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:32.789409    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f22efb3-18ab-43a2-a936-4a717371ece3" containerName="istio-proxy"
Jul 15 21:22:32.789455 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:32.789420    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc2b7d48-a365-4791-92a1-099a92df542f" containerName="revoke-keys"
Jul 15 21:22:32.789455 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:32.789426    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc2b7d48-a365-4791-92a1-099a92df542f" containerName="revoke-keys"
Jul 15 21:22:32.789455 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:32.789434    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5024762-4275-48ce-a7ff-43818cf49cf5" containerName="maas-api"
Jul 15 21:22:32.789455 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:32.789441    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5024762-4275-48ce-a7ff-43818cf49cf5" containerName="maas-api"
Jul 15 21:22:32.789455 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:32.789448    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f22efb3-18ab-43a2-a936-4a717371ece3" containerName="istio-proxy"
Jul 15 21:22:32.789455 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:32.789453    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f22efb3-18ab-43a2-a936-4a717371ece3" containerName="istio-proxy"
Jul 15 21:22:32.789709 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:32.789476    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb1054f6-f436-4f1b-a556-a6d4c394a44f" containerName="revoke-keys"
Jul 15 21:22:32.789709 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:32.789484    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb1054f6-f436-4f1b-a556-a6d4c394a44f" containerName="revoke-keys"
Jul 15 21:22:32.789709 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:32.789537    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f22efb3-18ab-43a2-a936-4a717371ece3" containerName="istio-proxy"
Jul 15 21:22:32.789709 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:32.789545    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5024762-4275-48ce-a7ff-43818cf49cf5" containerName="maas-api"
Jul 15 21:22:32.789709 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:32.789553    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc2b7d48-a365-4791-92a1-099a92df542f" containerName="revoke-keys"
Jul 15 21:22:32.789709 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:32.789559    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb1054f6-f436-4f1b-a556-a6d4c394a44f" containerName="revoke-keys"
Jul 15 21:22:32.792537 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:32.792515    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt"
Jul 15 21:22:32.795482 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:32.795453    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 15 21:22:32.795606 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:32.795486    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-a3485fb9-openshift-default-dockercfg-zpxx4\""
Jul 15 21:22:32.809388 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:32.809368    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:22:32.815771 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:32.815744    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt"]
Jul 15 21:22:32.849064 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:32.849028    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-istio-data\") pod \"e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt\" (UID: \"a5f8687c-0a6f-41f1-8c09-4f69e15d3731\") " pod="openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt"
Jul 15 21:22:32.849064 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:32.849065    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg9jq\" (UniqueName: \"kubernetes.io/projected/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-kube-api-access-xg9jq\") pod \"e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt\" (UID: \"a5f8687c-0a6f-41f1-8c09-4f69e15d3731\") " pod="openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt"
Jul 15 21:22:32.849317 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:32.849100    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-workload-socket\") pod \"e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt\" (UID: \"a5f8687c-0a6f-41f1-8c09-4f69e15d3731\") " pod="openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt"
Jul 15 21:22:32.849317 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:32.849177    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-istio-envoy\") pod \"e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt\" (UID: \"a5f8687c-0a6f-41f1-8c09-4f69e15d3731\") " pod="openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt"
Jul 15 21:22:32.849317 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:32.849229    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-istio-token\") pod \"e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt\" (UID: \"a5f8687c-0a6f-41f1-8c09-4f69e15d3731\") " pod="openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt"
Jul 15 21:22:32.849317 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:32.849255    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-credential-socket\") pod \"e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt\" (UID: \"a5f8687c-0a6f-41f1-8c09-4f69e15d3731\") " pod="openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt"
Jul 15 21:22:32.849317 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:32.849285    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-istiod-ca-cert\") pod \"e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt\" (UID: \"a5f8687c-0a6f-41f1-8c09-4f69e15d3731\") " pod="openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt"
Jul 15 21:22:32.849632 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:32.849327    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-istio-podinfo\") pod \"e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt\" (UID: \"a5f8687c-0a6f-41f1-8c09-4f69e15d3731\") " pod="openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt"
Jul 15 21:22:32.849632 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:32.849356    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-workload-certs\") pod \"e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt\" (UID: \"a5f8687c-0a6f-41f1-8c09-4f69e15d3731\") " pod="openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt"
Jul 15 21:22:32.950351 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:32.950311    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-istio-token\") pod \"e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt\" (UID: \"a5f8687c-0a6f-41f1-8c09-4f69e15d3731\") " pod="openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt"
Jul 15 21:22:32.950351 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:32.950351    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-credential-socket\") pod \"e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt\" (UID: \"a5f8687c-0a6f-41f1-8c09-4f69e15d3731\") " pod="openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt"
Jul 15 21:22:32.950626 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:32.950380    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-istiod-ca-cert\") pod \"e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt\" (UID: \"a5f8687c-0a6f-41f1-8c09-4f69e15d3731\") " pod="openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt"
Jul 15 21:22:32.950626 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:32.950404    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-istio-podinfo\") pod \"e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt\" (UID: \"a5f8687c-0a6f-41f1-8c09-4f69e15d3731\") " pod="openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt"
Jul 15 21:22:32.950626 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:32.950432    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-workload-certs\") pod \"e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt\" (UID: \"a5f8687c-0a6f-41f1-8c09-4f69e15d3731\") " pod="openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt"
Jul 15 21:22:32.950626 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:32.950476    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-istio-data\") pod \"e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt\" (UID: \"a5f8687c-0a6f-41f1-8c09-4f69e15d3731\") " pod="openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt"
Jul 15 21:22:32.950626 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:32.950502    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg9jq\" (UniqueName: \"kubernetes.io/projected/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-kube-api-access-xg9jq\") pod \"e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt\" (UID: \"a5f8687c-0a6f-41f1-8c09-4f69e15d3731\") " pod="openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt"
Jul 15 21:22:32.950626 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:32.950542    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-workload-socket\") pod \"e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt\" (UID: \"a5f8687c-0a6f-41f1-8c09-4f69e15d3731\") " pod="openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt"
Jul 15 21:22:32.950626 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:32.950583    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-istio-envoy\") pod \"e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt\" (UID: \"a5f8687c-0a6f-41f1-8c09-4f69e15d3731\") " pod="openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt"
Jul 15 21:22:32.951006 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:32.950839    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-credential-socket\") pod \"e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt\" (UID: \"a5f8687c-0a6f-41f1-8c09-4f69e15d3731\") " pod="openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt"
Jul 15 21:22:32.951053 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:32.951029    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-istio-data\") pod \"e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt\" (UID: \"a5f8687c-0a6f-41f1-8c09-4f69e15d3731\") " pod="openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt"
Jul 15 21:22:32.951103 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:32.951058    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-workload-certs\") pod \"e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt\" (UID: \"a5f8687c-0a6f-41f1-8c09-4f69e15d3731\") " pod="openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt"
Jul 15 21:22:32.951160 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:32.951102    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-workload-socket\") pod \"e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt\" (UID: \"a5f8687c-0a6f-41f1-8c09-4f69e15d3731\") " pod="openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt"
Jul 15 21:22:32.951210 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:32.951200    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-istiod-ca-cert\") pod \"e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt\" (UID: \"a5f8687c-0a6f-41f1-8c09-4f69e15d3731\") " pod="openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt"
Jul 15 21:22:32.952839 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:32.952821    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-istio-envoy\") pod \"e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt\" (UID: \"a5f8687c-0a6f-41f1-8c09-4f69e15d3731\") " pod="openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt"
Jul 15 21:22:32.952963 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:32.952945    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-istio-podinfo\") pod \"e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt\" (UID: \"a5f8687c-0a6f-41f1-8c09-4f69e15d3731\") " pod="openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt"
Jul 15 21:22:32.962707 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:32.962685    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg9jq\" (UniqueName: \"kubernetes.io/projected/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-kube-api-access-xg9jq\") pod \"e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt\" (UID: \"a5f8687c-0a6f-41f1-8c09-4f69e15d3731\") " pod="openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt"
Jul 15 21:22:32.962852 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:32.962831    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-istio-token\") pod \"e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt\" (UID: \"a5f8687c-0a6f-41f1-8c09-4f69e15d3731\") " pod="openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt"
Jul 15 21:22:33.008470 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:33.008433    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:22:33.104065 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:33.103978    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt"
Jul 15 21:22:33.246652 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:33.246630    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt"]
Jul 15 21:22:33.248682 ip-10-0-137-234 kubenswrapper[2566]: W0715 21:22:33.248654    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5f8687c_0a6f_41f1_8c09_4f69e15d3731.slice/crio-6037e9fd4f4fbe96858afeb569dbb21b1c7fb52ddb19aa04d944ad1b4c8aed68 WatchSource:0}: Error finding container 6037e9fd4f4fbe96858afeb569dbb21b1c7fb52ddb19aa04d944ad1b4c8aed68: Status 404 returned error can't find the container with id 6037e9fd4f4fbe96858afeb569dbb21b1c7fb52ddb19aa04d944ad1b4c8aed68
Jul 15 21:22:33.250580 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:33.250548    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:22:33.250644 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:33.250610    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:22:33.250687 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:33.250641    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:22:34.055945 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:34.055910    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt" event={"ID":"a5f8687c-0a6f-41f1-8c09-4f69e15d3731","Type":"ContainerStarted","Data":"3217ab55db8d63c4c6c1d9b86356ea73da7059bc08f370370566caa360d8ec69"}
Jul 15 21:22:34.056133 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:34.055961    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt" event={"ID":"a5f8687c-0a6f-41f1-8c09-4f69e15d3731","Type":"ContainerStarted","Data":"6037e9fd4f4fbe96858afeb569dbb21b1c7fb52ddb19aa04d944ad1b4c8aed68"}
Jul 15 21:22:34.096392 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:34.096342    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt" podStartSLOduration=2.096325723 podStartE2EDuration="2.096325723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:22:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:22:34.093232413 +0000 UTC m=+2087.242270481" watchObservedRunningTime="2026-07-15 21:22:34.096325723 +0000 UTC m=+2087.245363779"
Jul 15 21:22:34.104560 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:34.104526    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt"
Jul 15 21:22:34.109441 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:34.109415    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt"
Jul 15 21:22:35.060955 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:35.060924    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt"
Jul 15 21:22:35.061998 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:35.061976    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt"
Jul 15 21:22:36.188822 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:36.188776    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:22:36.296299 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:36.296265    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:22:36.318643 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:36.318611    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a3485fb9-776f4fd668-vs8kv"]
Jul 15 21:22:36.319288 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:36.319268    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f22efb3-18ab-43a2-a936-4a717371ece3" containerName="istio-proxy"
Jul 15 21:22:36.322812 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:36.322792    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3485fb9-776f4fd668-vs8kv"
Jul 15 21:22:36.326188 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:36.326167    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-a3485fb9-serving-cert\""
Jul 15 21:22:36.326493 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:36.326477    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 15 21:22:36.327762 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:36.327742    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-s2l9d\""
Jul 15 21:22:36.327857 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:36.327760    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 15 21:22:36.335204 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:36.335174    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a3485fb9-776f4fd668-vs8kv"]
Jul 15 21:22:36.382214 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:36.382165    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b735b071-7f94-4f14-bf7c-c88216f170e4-maas-api-tls\") pod \"maas-api-e2e-mt-a3485fb9-776f4fd668-vs8kv\" (UID: \"b735b071-7f94-4f14-bf7c-c88216f170e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3485fb9-776f4fd668-vs8kv"
Jul 15 21:22:36.382214 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:36.382212    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b735b071-7f94-4f14-bf7c-c88216f170e4-openshift-service-ca\") pod \"maas-api-e2e-mt-a3485fb9-776f4fd668-vs8kv\" (UID: \"b735b071-7f94-4f14-bf7c-c88216f170e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3485fb9-776f4fd668-vs8kv"
Jul 15 21:22:36.382489 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:36.382240    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nxhd\" (UniqueName: \"kubernetes.io/projected/b735b071-7f94-4f14-bf7c-c88216f170e4-kube-api-access-5nxhd\") pod \"maas-api-e2e-mt-a3485fb9-776f4fd668-vs8kv\" (UID: \"b735b071-7f94-4f14-bf7c-c88216f170e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3485fb9-776f4fd668-vs8kv"
Jul 15 21:22:36.395220 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:36.395190    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:22:36.483588 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:36.483548    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b735b071-7f94-4f14-bf7c-c88216f170e4-maas-api-tls\") pod \"maas-api-e2e-mt-a3485fb9-776f4fd668-vs8kv\" (UID: \"b735b071-7f94-4f14-bf7c-c88216f170e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3485fb9-776f4fd668-vs8kv"
Jul 15 21:22:36.483770 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:36.483601    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b735b071-7f94-4f14-bf7c-c88216f170e4-openshift-service-ca\") pod \"maas-api-e2e-mt-a3485fb9-776f4fd668-vs8kv\" (UID: \"b735b071-7f94-4f14-bf7c-c88216f170e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3485fb9-776f4fd668-vs8kv"
Jul 15 21:22:36.483770 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:36.483630    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nxhd\" (UniqueName: \"kubernetes.io/projected/b735b071-7f94-4f14-bf7c-c88216f170e4-kube-api-access-5nxhd\") pod \"maas-api-e2e-mt-a3485fb9-776f4fd668-vs8kv\" (UID: \"b735b071-7f94-4f14-bf7c-c88216f170e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3485fb9-776f4fd668-vs8kv"
Jul 15 21:22:36.483770 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:22:36.483707    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-a3485fb9-serving-cert: secret "maas-api-e2e-mt-a3485fb9-serving-cert" not found
Jul 15 21:22:36.483994 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:22:36.483781    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b735b071-7f94-4f14-bf7c-c88216f170e4-maas-api-tls podName:b735b071-7f94-4f14-bf7c-c88216f170e4 nodeName:}" failed. No retries permitted until 2026-07-15 21:22:36.983756177 +0000 UTC m=+2090.132794225 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b735b071-7f94-4f14-bf7c-c88216f170e4-maas-api-tls") pod "maas-api-e2e-mt-a3485fb9-776f4fd668-vs8kv" (UID: "b735b071-7f94-4f14-bf7c-c88216f170e4") : secret "maas-api-e2e-mt-a3485fb9-serving-cert" not found
Jul 15 21:22:36.484329 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:36.484305    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b735b071-7f94-4f14-bf7c-c88216f170e4-openshift-service-ca\") pod \"maas-api-e2e-mt-a3485fb9-776f4fd668-vs8kv\" (UID: \"b735b071-7f94-4f14-bf7c-c88216f170e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3485fb9-776f4fd668-vs8kv"
Jul 15 21:22:36.498050 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:36.498029    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nxhd\" (UniqueName: \"kubernetes.io/projected/b735b071-7f94-4f14-bf7c-c88216f170e4-kube-api-access-5nxhd\") pod \"maas-api-e2e-mt-a3485fb9-776f4fd668-vs8kv\" (UID: \"b735b071-7f94-4f14-bf7c-c88216f170e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3485fb9-776f4fd668-vs8kv"
Jul 15 21:22:36.807586 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:36.807496    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:22:36.988050 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:36.988014    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b735b071-7f94-4f14-bf7c-c88216f170e4-maas-api-tls\") pod \"maas-api-e2e-mt-a3485fb9-776f4fd668-vs8kv\" (UID: \"b735b071-7f94-4f14-bf7c-c88216f170e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3485fb9-776f4fd668-vs8kv"
Jul 15 21:22:36.990545 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:36.990518    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b735b071-7f94-4f14-bf7c-c88216f170e4-maas-api-tls\") pod \"maas-api-e2e-mt-a3485fb9-776f4fd668-vs8kv\" (UID: \"b735b071-7f94-4f14-bf7c-c88216f170e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3485fb9-776f4fd668-vs8kv"
Jul 15 21:22:37.003675 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:37.003634    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:22:37.098191 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:37.098099    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:22:37.234587 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:37.234543    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3485fb9-776f4fd668-vs8kv"
Jul 15 21:22:37.387571 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:37.387543    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a3485fb9-776f4fd668-vs8kv"]
Jul 15 21:22:37.391857 ip-10-0-137-234 kubenswrapper[2566]: W0715 21:22:37.391825    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb735b071_7f94_4f14_bf7c_c88216f170e4.slice/crio-e22494b262330a9f80a3eed53833323909cd7f166699f6148e020cd0a7f94411 WatchSource:0}: Error finding container e22494b262330a9f80a3eed53833323909cd7f166699f6148e020cd0a7f94411: Status 404 returned error can't find the container with id e22494b262330a9f80a3eed53833323909cd7f166699f6148e020cd0a7f94411
Jul 15 21:22:37.393144 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:37.393125    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 15 21:22:38.081759 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:38.081718    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3485fb9-776f4fd668-vs8kv" event={"ID":"b735b071-7f94-4f14-bf7c-c88216f170e4","Type":"ContainerStarted","Data":"095bafde4bd7b738780fec744ff6da03e7d57a1412279109ab3aea8ecdb21400"}
Jul 15 21:22:38.081963 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:38.081821    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3485fb9-776f4fd668-vs8kv"
Jul 15 21:22:38.081963 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:38.081848    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3485fb9-776f4fd668-vs8kv" event={"ID":"b735b071-7f94-4f14-bf7c-c88216f170e4","Type":"ContainerStarted","Data":"e22494b262330a9f80a3eed53833323909cd7f166699f6148e020cd0a7f94411"}
Jul 15 21:22:38.109342 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:38.109283    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3485fb9-776f4fd668-vs8kv" podStartSLOduration=1.517004131 podStartE2EDuration="2.109266559s" totalImagesPullingTime="592.262428ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:22:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:22:38.109118378 +0000 UTC m=+2091.258156454" watchObservedRunningTime="2026-07-15 21:22:38.109266559 +0000 UTC m=+2091.258304618"
Jul 15 21:22:40.730133 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:40.730093    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:22:41.434884 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:41.434846    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:22:41.531063 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:41.531029    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:22:41.930134 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:41.930101    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:22:44.098203 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:44.098161    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3485fb9-776f4fd668-vs8kv"
Jul 15 21:22:47.799264 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:47.799229    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-652lk_f9a159bb-39c4-4cca-92ff-1c74d739e311/ovn-acl-logging/0.log"
Jul 15 21:22:47.801254 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:47.801232    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-652lk_f9a159bb-39c4-4cca-92ff-1c74d739e311/ovn-acl-logging/0.log"
Jul 15 21:22:48.301119 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:48.301086    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:22:48.397637 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:48.397607    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:22:48.700534 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:48.700498    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a3485fb9-rg8jc"]
Jul 15 21:22:48.705833 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:48.705803    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a3485fb9-rg8jc"
Jul 15 21:22:48.709838 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:48.709815    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zsrf5\""
Jul 15 21:22:48.715477 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:48.715452    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a3485fb9-rg8jc"]
Jul 15 21:22:48.790124 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:48.790081    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4be6efe-444a-47b9-96d2-4896b8f91e82-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a3485fb9-rg8jc\" (UID: \"a4be6efe-444a-47b9-96d2-4896b8f91e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a3485fb9-rg8jc"
Jul 15 21:22:48.891110 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:48.891074    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4be6efe-444a-47b9-96d2-4896b8f91e82-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a3485fb9-rg8jc\" (UID: \"a4be6efe-444a-47b9-96d2-4896b8f91e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a3485fb9-rg8jc"
Jul 15 21:22:48.891742 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:48.891722    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4be6efe-444a-47b9-96d2-4896b8f91e82-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a3485fb9-rg8jc\" (UID: \"a4be6efe-444a-47b9-96d2-4896b8f91e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a3485fb9-rg8jc"
Jul 15 21:22:49.018068 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:49.017979    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a3485fb9-rg8jc"
Jul 15 21:22:49.142097 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:49.142042    2566 generic.go:356] "Generic (PLEG): container finished" podID="a5f8687c-0a6f-41f1-8c09-4f69e15d3731" containerID="3217ab55db8d63c4c6c1d9b86356ea73da7059bc08f370370566caa360d8ec69" exitCode=0
Jul 15 21:22:49.142097 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:49.142088    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt" event={"ID":"a5f8687c-0a6f-41f1-8c09-4f69e15d3731","Type":"ContainerDied","Data":"3217ab55db8d63c4c6c1d9b86356ea73da7059bc08f370370566caa360d8ec69"}
Jul 15 21:22:49.142557 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:49.142539    2566 scope.go:117] "RemoveContainer" containerID="3217ab55db8d63c4c6c1d9b86356ea73da7059bc08f370370566caa360d8ec69"
Jul 15 21:22:49.143612 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:49.143582    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:22:49.143700 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:49.143675    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:22:49.143743 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:49.143717    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:22:49.160627 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:49.160588    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a3485fb9-rg8jc"]
Jul 15 21:22:49.162732 ip-10-0-137-234 kubenswrapper[2566]: W0715 21:22:49.162708    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4be6efe_444a_47b9_96d2_4896b8f91e82.slice/crio-d674c28e0d71a45c81ff1bc9e8f054a531344291663e90186d825f799085db73 WatchSource:0}: Error finding container d674c28e0d71a45c81ff1bc9e8f054a531344291663e90186d825f799085db73: Status 404 returned error can't find the container with id d674c28e0d71a45c81ff1bc9e8f054a531344291663e90186d825f799085db73
Jul 15 21:22:50.061631 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:50.061600    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt"
Jul 15 21:22:50.148032 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:50.147997    2566 generic.go:356] "Generic (PLEG): container finished" podID="a4be6efe-444a-47b9-96d2-4896b8f91e82" containerID="406b27d285a3338dea03c100382291c10db0bba760249bf61ea0ace30ff87ef7" exitCode=0
Jul 15 21:22:50.148212 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:50.148076    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a3485fb9-rg8jc" event={"ID":"a4be6efe-444a-47b9-96d2-4896b8f91e82","Type":"ContainerDied","Data":"406b27d285a3338dea03c100382291c10db0bba760249bf61ea0ace30ff87ef7"}
Jul 15 21:22:50.148212 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:50.148130    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a3485fb9-rg8jc" event={"ID":"a4be6efe-444a-47b9-96d2-4896b8f91e82","Type":"ContainerStarted","Data":"d674c28e0d71a45c81ff1bc9e8f054a531344291663e90186d825f799085db73"}
Jul 15 21:22:50.149886 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:50.149863    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt" event={"ID":"a5f8687c-0a6f-41f1-8c09-4f69e15d3731","Type":"ContainerStarted","Data":"e9eba6c39467bfcc5b0304b6247069303ce1a42cbbb7e975cc2609a43a2e4024"}
Jul 15 21:22:51.104337 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:51.104299    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt"
Jul 15 21:22:51.105744 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:51.105710    2566 patch_prober.go:28] interesting pod/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Jul 15 21:22:51.105881 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:51.105768    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt" podUID="a5f8687c-0a6f-41f1-8c09-4f69e15d3731" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Jul 15 21:22:51.326447 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:51.326422    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a3485fb9-rg8jc"
Jul 15 21:22:51.415460 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:51.415368    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4be6efe-444a-47b9-96d2-4896b8f91e82-maas-api-service-ca\") pod \"a4be6efe-444a-47b9-96d2-4896b8f91e82\" (UID: \"a4be6efe-444a-47b9-96d2-4896b8f91e82\") "
Jul 15 21:22:51.415761 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:51.415733    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4be6efe-444a-47b9-96d2-4896b8f91e82-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a4be6efe-444a-47b9-96d2-4896b8f91e82" (UID: "a4be6efe-444a-47b9-96d2-4896b8f91e82"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 21:22:51.516829 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:51.516789    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4be6efe-444a-47b9-96d2-4896b8f91e82-maas-api-service-ca\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:22:52.105373 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:52.105283    2566 patch_prober.go:28] interesting pod/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Jul 15 21:22:52.105373 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:52.105341    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt" podUID="a5f8687c-0a6f-41f1-8c09-4f69e15d3731" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Jul 15 21:22:52.161577 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:52.161551    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a3485fb9-rg8jc"
Jul 15 21:22:52.161740 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:52.161590    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a3485fb9-rg8jc" event={"ID":"a4be6efe-444a-47b9-96d2-4896b8f91e82","Type":"ContainerDied","Data":"d674c28e0d71a45c81ff1bc9e8f054a531344291663e90186d825f799085db73"}
Jul 15 21:22:52.161740 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:52.161639    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d674c28e0d71a45c81ff1bc9e8f054a531344291663e90186d825f799085db73"
Jul 15 21:22:53.104522 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:53.104481    2566 patch_prober.go:28] interesting pod/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Jul 15 21:22:53.104716 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:53.104549    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt" podUID="a5f8687c-0a6f-41f1-8c09-4f69e15d3731" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Jul 15 21:22:54.109150 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:54.109122    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt"
Jul 15 21:22:54.109521 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:54.109430    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt"
Jul 15 21:22:54.110027 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:54.110011    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt"
Jul 15 21:22:56.444126 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:56.444090    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:22:56.544684 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:56.544622    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:22:59.293099 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:59.293060    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:22:59.399692 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:59.399653    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:22:59.580130 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:59.580049    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a3485fb9-776f4fd668-vs8kv"]
Jul 15 21:22:59.580360 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:59.580331    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3485fb9-776f4fd668-vs8kv" podUID="b735b071-7f94-4f14-bf7c-c88216f170e4" containerName="maas-api" containerID="cri-o://095bafde4bd7b738780fec744ff6da03e7d57a1412279109ab3aea8ecdb21400" gracePeriod=30
Jul 15 21:22:59.617013 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:59.616982    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:22:59.832426 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:59.832365    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3485fb9-776f4fd668-vs8kv"
Jul 15 21:22:59.997135 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:59.997100    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b735b071-7f94-4f14-bf7c-c88216f170e4-maas-api-tls\") pod \"b735b071-7f94-4f14-bf7c-c88216f170e4\" (UID: \"b735b071-7f94-4f14-bf7c-c88216f170e4\") "
Jul 15 21:22:59.997295 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:59.997181    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b735b071-7f94-4f14-bf7c-c88216f170e4-openshift-service-ca\") pod \"b735b071-7f94-4f14-bf7c-c88216f170e4\" (UID: \"b735b071-7f94-4f14-bf7c-c88216f170e4\") "
Jul 15 21:22:59.997295 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:59.997214    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5nxhd\" (UniqueName: \"kubernetes.io/projected/b735b071-7f94-4f14-bf7c-c88216f170e4-kube-api-access-5nxhd\") pod \"b735b071-7f94-4f14-bf7c-c88216f170e4\" (UID: \"b735b071-7f94-4f14-bf7c-c88216f170e4\") "
Jul 15 21:22:59.997614 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:59.997586    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b735b071-7f94-4f14-bf7c-c88216f170e4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b735b071-7f94-4f14-bf7c-c88216f170e4" (UID: "b735b071-7f94-4f14-bf7c-c88216f170e4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 21:22:59.999242 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:59.999213    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b735b071-7f94-4f14-bf7c-c88216f170e4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b735b071-7f94-4f14-bf7c-c88216f170e4" (UID: "b735b071-7f94-4f14-bf7c-c88216f170e4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 21:22:59.999338 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:22:59.999267    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b735b071-7f94-4f14-bf7c-c88216f170e4-kube-api-access-5nxhd" (OuterVolumeSpecName: "kube-api-access-5nxhd") pod "b735b071-7f94-4f14-bf7c-c88216f170e4" (UID: "b735b071-7f94-4f14-bf7c-c88216f170e4"). InnerVolumeSpecName "kube-api-access-5nxhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 21:23:00.098035 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:00.097951    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b735b071-7f94-4f14-bf7c-c88216f170e4-openshift-service-ca\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:23:00.098035 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:00.097981    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nxhd\" (UniqueName: \"kubernetes.io/projected/b735b071-7f94-4f14-bf7c-c88216f170e4-kube-api-access-5nxhd\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:23:00.098035 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:00.097992    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b735b071-7f94-4f14-bf7c-c88216f170e4-maas-api-tls\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:23:00.201174 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:00.201133    2566 generic.go:356] "Generic (PLEG): container finished" podID="b735b071-7f94-4f14-bf7c-c88216f170e4" containerID="095bafde4bd7b738780fec744ff6da03e7d57a1412279109ab3aea8ecdb21400" exitCode=0
Jul 15 21:23:00.201340 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:00.201182    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3485fb9-776f4fd668-vs8kv"
Jul 15 21:23:00.201340 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:00.201217    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3485fb9-776f4fd668-vs8kv" event={"ID":"b735b071-7f94-4f14-bf7c-c88216f170e4","Type":"ContainerDied","Data":"095bafde4bd7b738780fec744ff6da03e7d57a1412279109ab3aea8ecdb21400"}
Jul 15 21:23:00.201340 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:00.201278    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3485fb9-776f4fd668-vs8kv" event={"ID":"b735b071-7f94-4f14-bf7c-c88216f170e4","Type":"ContainerDied","Data":"e22494b262330a9f80a3eed53833323909cd7f166699f6148e020cd0a7f94411"}
Jul 15 21:23:00.201340 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:00.201300    2566 scope.go:117] "RemoveContainer" containerID="095bafde4bd7b738780fec744ff6da03e7d57a1412279109ab3aea8ecdb21400"
Jul 15 21:23:00.210676 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:00.210658    2566 scope.go:117] "RemoveContainer" containerID="095bafde4bd7b738780fec744ff6da03e7d57a1412279109ab3aea8ecdb21400"
Jul 15 21:23:00.210980 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:23:00.210959    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"095bafde4bd7b738780fec744ff6da03e7d57a1412279109ab3aea8ecdb21400\": container with ID starting with 095bafde4bd7b738780fec744ff6da03e7d57a1412279109ab3aea8ecdb21400 not found: ID does not exist" containerID="095bafde4bd7b738780fec744ff6da03e7d57a1412279109ab3aea8ecdb21400"
Jul 15 21:23:00.211063 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:00.210991    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"095bafde4bd7b738780fec744ff6da03e7d57a1412279109ab3aea8ecdb21400"} err="failed to get container status \"095bafde4bd7b738780fec744ff6da03e7d57a1412279109ab3aea8ecdb21400\": rpc error: code = NotFound desc = could not find container \"095bafde4bd7b738780fec744ff6da03e7d57a1412279109ab3aea8ecdb21400\": container with ID starting with 095bafde4bd7b738780fec744ff6da03e7d57a1412279109ab3aea8ecdb21400 not found: ID does not exist"
Jul 15 21:23:00.225868 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:00.225840    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a3485fb9-776f4fd668-vs8kv"]
Jul 15 21:23:00.234497 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:00.234470    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a3485fb9-776f4fd668-vs8kv"]
Jul 15 21:23:01.756553 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:01.756517    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b735b071-7f94-4f14-bf7c-c88216f170e4" path="/var/lib/kubelet/pods/b735b071-7f94-4f14-bf7c-c88216f170e4/volumes"
Jul 15 21:23:03.803780 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:03.803732    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:23:10.316204 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:10.316162    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:23:10.390565 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:10.390529    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:23:14.560831 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:14.560796    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt"]
Jul 15 21:23:14.561434 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:14.561085    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt" podUID="a5f8687c-0a6f-41f1-8c09-4f69e15d3731" containerName="istio-proxy" containerID="cri-o://e9eba6c39467bfcc5b0304b6247069303ce1a42cbbb7e975cc2609a43a2e4024" gracePeriod=30
Jul 15 21:23:14.604099 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:14.604062    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:23:14.848147 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:14.843209    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:23:14.975427 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:14.975392    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:23:15.088684 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:15.088646    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:23:15.244694 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:15.244666    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:23:19.810923 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:19.810879    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt"
Jul 15 21:23:19.861521 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:19.861440    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-istio-token\") pod \"a5f8687c-0a6f-41f1-8c09-4f69e15d3731\" (UID: \"a5f8687c-0a6f-41f1-8c09-4f69e15d3731\") "
Jul 15 21:23:19.861521 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:19.861476    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-istio-envoy\") pod \"a5f8687c-0a6f-41f1-8c09-4f69e15d3731\" (UID: \"a5f8687c-0a6f-41f1-8c09-4f69e15d3731\") "
Jul 15 21:23:19.861521 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:19.861496    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-istio-podinfo\") pod \"a5f8687c-0a6f-41f1-8c09-4f69e15d3731\" (UID: \"a5f8687c-0a6f-41f1-8c09-4f69e15d3731\") "
Jul 15 21:23:19.861521 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:19.861518    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-istio-data\") pod \"a5f8687c-0a6f-41f1-8c09-4f69e15d3731\" (UID: \"a5f8687c-0a6f-41f1-8c09-4f69e15d3731\") "
Jul 15 21:23:19.861843 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:19.861539    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-workload-socket\") pod \"a5f8687c-0a6f-41f1-8c09-4f69e15d3731\" (UID: \"a5f8687c-0a6f-41f1-8c09-4f69e15d3731\") "
Jul 15 21:23:19.861843 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:19.861575    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-istiod-ca-cert\") pod \"a5f8687c-0a6f-41f1-8c09-4f69e15d3731\" (UID: \"a5f8687c-0a6f-41f1-8c09-4f69e15d3731\") "
Jul 15 21:23:19.861843 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:19.861627    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xg9jq\" (UniqueName: \"kubernetes.io/projected/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-kube-api-access-xg9jq\") pod \"a5f8687c-0a6f-41f1-8c09-4f69e15d3731\" (UID: \"a5f8687c-0a6f-41f1-8c09-4f69e15d3731\") "
Jul 15 21:23:19.862053 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:19.861882    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a5f8687c-0a6f-41f1-8c09-4f69e15d3731" (UID: "a5f8687c-0a6f-41f1-8c09-4f69e15d3731"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:23:19.862053 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:19.862011    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a5f8687c-0a6f-41f1-8c09-4f69e15d3731" (UID: "a5f8687c-0a6f-41f1-8c09-4f69e15d3731"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 21:23:19.862157 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:19.862059    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-workload-certs\") pod \"a5f8687c-0a6f-41f1-8c09-4f69e15d3731\" (UID: \"a5f8687c-0a6f-41f1-8c09-4f69e15d3731\") "
Jul 15 21:23:19.862157 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:19.862123    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-credential-socket\") pod \"a5f8687c-0a6f-41f1-8c09-4f69e15d3731\" (UID: \"a5f8687c-0a6f-41f1-8c09-4f69e15d3731\") "
Jul 15 21:23:19.862516 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:19.862462    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a5f8687c-0a6f-41f1-8c09-4f69e15d3731" (UID: "a5f8687c-0a6f-41f1-8c09-4f69e15d3731"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:23:19.862649 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:19.862473    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-istio-data" (OuterVolumeSpecName: "istio-data") pod "a5f8687c-0a6f-41f1-8c09-4f69e15d3731" (UID: "a5f8687c-0a6f-41f1-8c09-4f69e15d3731"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:23:19.862649 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:19.862560    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a5f8687c-0a6f-41f1-8c09-4f69e15d3731" (UID: "a5f8687c-0a6f-41f1-8c09-4f69e15d3731"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:23:19.863118 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:19.863094    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-workload-socket\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:23:19.863236 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:19.863121    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-istiod-ca-cert\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:23:19.863236 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:19.863136    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-workload-certs\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:23:19.863236 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:19.863149    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-credential-socket\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:23:19.863236 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:19.863162    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-istio-data\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:23:19.864037 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:19.864013    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a5f8687c-0a6f-41f1-8c09-4f69e15d3731" (UID: "a5f8687c-0a6f-41f1-8c09-4f69e15d3731"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:23:19.864253 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:19.864234    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a5f8687c-0a6f-41f1-8c09-4f69e15d3731" (UID: "a5f8687c-0a6f-41f1-8c09-4f69e15d3731"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 15 21:23:19.864456 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:19.864439    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-istio-token" (OuterVolumeSpecName: "istio-token") pod "a5f8687c-0a6f-41f1-8c09-4f69e15d3731" (UID: "a5f8687c-0a6f-41f1-8c09-4f69e15d3731"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 21:23:19.864584 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:19.864560    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-kube-api-access-xg9jq" (OuterVolumeSpecName: "kube-api-access-xg9jq") pod "a5f8687c-0a6f-41f1-8c09-4f69e15d3731" (UID: "a5f8687c-0a6f-41f1-8c09-4f69e15d3731"). InnerVolumeSpecName "kube-api-access-xg9jq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 21:23:19.964284 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:19.964248    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg9jq\" (UniqueName: \"kubernetes.io/projected/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-kube-api-access-xg9jq\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:23:19.964284 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:19.964280    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-istio-token\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:23:19.964284 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:19.964289    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-istio-envoy\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:23:19.964501 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:19.964298    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f8687c-0a6f-41f1-8c09-4f69e15d3731-istio-podinfo\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:23:20.305037 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:20.304999    2566 generic.go:356] "Generic (PLEG): container finished" podID="a5f8687c-0a6f-41f1-8c09-4f69e15d3731" containerID="e9eba6c39467bfcc5b0304b6247069303ce1a42cbbb7e975cc2609a43a2e4024" exitCode=0
Jul 15 21:23:20.305211 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:20.305062    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt"
Jul 15 21:23:20.305211 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:20.305084    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt" event={"ID":"a5f8687c-0a6f-41f1-8c09-4f69e15d3731","Type":"ContainerDied","Data":"e9eba6c39467bfcc5b0304b6247069303ce1a42cbbb7e975cc2609a43a2e4024"}
Jul 15 21:23:20.305211 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:20.305177    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt" event={"ID":"a5f8687c-0a6f-41f1-8c09-4f69e15d3731","Type":"ContainerDied","Data":"6037e9fd4f4fbe96858afeb569dbb21b1c7fb52ddb19aa04d944ad1b4c8aed68"}
Jul 15 21:23:20.305211 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:20.305206    2566 scope.go:117] "RemoveContainer" containerID="e9eba6c39467bfcc5b0304b6247069303ce1a42cbbb7e975cc2609a43a2e4024"
Jul 15 21:23:20.314499 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:20.314481    2566 scope.go:117] "RemoveContainer" containerID="3217ab55db8d63c4c6c1d9b86356ea73da7059bc08f370370566caa360d8ec69"
Jul 15 21:23:20.322473 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:20.322456    2566 scope.go:117] "RemoveContainer" containerID="e9eba6c39467bfcc5b0304b6247069303ce1a42cbbb7e975cc2609a43a2e4024"
Jul 15 21:23:20.322706 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:23:20.322685    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9eba6c39467bfcc5b0304b6247069303ce1a42cbbb7e975cc2609a43a2e4024\": container with ID starting with e9eba6c39467bfcc5b0304b6247069303ce1a42cbbb7e975cc2609a43a2e4024 not found: ID does not exist" containerID="e9eba6c39467bfcc5b0304b6247069303ce1a42cbbb7e975cc2609a43a2e4024"
Jul 15 21:23:20.322749 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:20.322715    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9eba6c39467bfcc5b0304b6247069303ce1a42cbbb7e975cc2609a43a2e4024"} err="failed to get container status \"e9eba6c39467bfcc5b0304b6247069303ce1a42cbbb7e975cc2609a43a2e4024\": rpc error: code = NotFound desc = could not find container \"e9eba6c39467bfcc5b0304b6247069303ce1a42cbbb7e975cc2609a43a2e4024\": container with ID starting with e9eba6c39467bfcc5b0304b6247069303ce1a42cbbb7e975cc2609a43a2e4024 not found: ID does not exist"
Jul 15 21:23:20.322749 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:20.322732    2566 scope.go:117] "RemoveContainer" containerID="3217ab55db8d63c4c6c1d9b86356ea73da7059bc08f370370566caa360d8ec69"
Jul 15 21:23:20.323075 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:23:20.323058    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3217ab55db8d63c4c6c1d9b86356ea73da7059bc08f370370566caa360d8ec69\": container with ID starting with 3217ab55db8d63c4c6c1d9b86356ea73da7059bc08f370370566caa360d8ec69 not found: ID does not exist" containerID="3217ab55db8d63c4c6c1d9b86356ea73da7059bc08f370370566caa360d8ec69"
Jul 15 21:23:20.323121 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:20.323080    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3217ab55db8d63c4c6c1d9b86356ea73da7059bc08f370370566caa360d8ec69"} err="failed to get container status \"3217ab55db8d63c4c6c1d9b86356ea73da7059bc08f370370566caa360d8ec69\": rpc error: code = NotFound desc = could not find container \"3217ab55db8d63c4c6c1d9b86356ea73da7059bc08f370370566caa360d8ec69\": container with ID starting with 3217ab55db8d63c4c6c1d9b86356ea73da7059bc08f370370566caa360d8ec69 not found: ID does not exist"
Jul 15 21:23:20.331480 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:20.331458    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt"]
Jul 15 21:23:20.335534 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:20.335514    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a3485fb9-openshift-default-5cfdc7854c-hw2nt"]
Jul 15 21:23:21.674867 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:21.674832    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:23:21.756197 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:21.756166    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5f8687c-0a6f-41f1-8c09-4f69e15d3731" path="/var/lib/kubelet/pods/a5f8687c-0a6f-41f1-8c09-4f69e15d3731/volumes"
Jul 15 21:23:21.780709 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:21.780673    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:23:21.882803 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:21.882773    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:23:22.088732 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:22.088698    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:23:28.408993 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:28.408958    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:23:28.504912 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:28.504858    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:23:35.417919 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:35.417850    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:23:35.738558 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:35.738524    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz"]
Jul 15 21:23:35.738921 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:35.738888    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5f8687c-0a6f-41f1-8c09-4f69e15d3731" containerName="istio-proxy"
Jul 15 21:23:35.738921 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:35.738919    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5f8687c-0a6f-41f1-8c09-4f69e15d3731" containerName="istio-proxy"
Jul 15 21:23:35.739005 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:35.738936    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b735b071-7f94-4f14-bf7c-c88216f170e4" containerName="maas-api"
Jul 15 21:23:35.739005 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:35.738942    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b735b071-7f94-4f14-bf7c-c88216f170e4" containerName="maas-api"
Jul 15 21:23:35.739005 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:35.738967    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4be6efe-444a-47b9-96d2-4896b8f91e82" containerName="revoke-keys"
Jul 15 21:23:35.739005 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:35.738974    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4be6efe-444a-47b9-96d2-4896b8f91e82" containerName="revoke-keys"
Jul 15 21:23:35.739129 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:35.739035    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5f8687c-0a6f-41f1-8c09-4f69e15d3731" containerName="istio-proxy"
Jul 15 21:23:35.739129 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:35.739046    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5f8687c-0a6f-41f1-8c09-4f69e15d3731" containerName="istio-proxy"
Jul 15 21:23:35.739129 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:35.739053    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4be6efe-444a-47b9-96d2-4896b8f91e82" containerName="revoke-keys"
Jul 15 21:23:35.739129 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:35.739061    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b735b071-7f94-4f14-bf7c-c88216f170e4" containerName="maas-api"
Jul 15 21:23:35.741964 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:35.741944    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz"
Jul 15 21:23:35.744920 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:35.744876    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-61280292-openshift-default-dockercfg-qjqcs\""
Jul 15 21:23:35.745204 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:35.745179    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 15 21:23:35.757823 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:35.757795    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz"]
Jul 15 21:23:35.766350 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:35.766312    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:23:35.802203 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:35.802165    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/903ced98-26ba-41e4-a7f7-306364f2bbba-istiod-ca-cert\") pod \"e2e-mt-61280292-openshift-default-59745789f-4wmpz\" (UID: \"903ced98-26ba-41e4-a7f7-306364f2bbba\") " pod="openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz"
Jul 15 21:23:35.802385 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:35.802225    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/903ced98-26ba-41e4-a7f7-306364f2bbba-istio-token\") pod \"e2e-mt-61280292-openshift-default-59745789f-4wmpz\" (UID: \"903ced98-26ba-41e4-a7f7-306364f2bbba\") " pod="openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz"
Jul 15 21:23:35.802385 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:35.802294    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/903ced98-26ba-41e4-a7f7-306364f2bbba-workload-socket\") pod \"e2e-mt-61280292-openshift-default-59745789f-4wmpz\" (UID: \"903ced98-26ba-41e4-a7f7-306364f2bbba\") " pod="openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz"
Jul 15 21:23:35.802385 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:35.802376    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/903ced98-26ba-41e4-a7f7-306364f2bbba-workload-certs\") pod \"e2e-mt-61280292-openshift-default-59745789f-4wmpz\" (UID: \"903ced98-26ba-41e4-a7f7-306364f2bbba\") " pod="openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz"
Jul 15 21:23:35.802519 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:35.802420    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/903ced98-26ba-41e4-a7f7-306364f2bbba-istio-envoy\") pod \"e2e-mt-61280292-openshift-default-59745789f-4wmpz\" (UID: \"903ced98-26ba-41e4-a7f7-306364f2bbba\") " pod="openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz"
Jul 15 21:23:35.802519 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:35.802438    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/903ced98-26ba-41e4-a7f7-306364f2bbba-istio-podinfo\") pod \"e2e-mt-61280292-openshift-default-59745789f-4wmpz\" (UID: \"903ced98-26ba-41e4-a7f7-306364f2bbba\") " pod="openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz"
Jul 15 21:23:35.802519 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:35.802477    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/903ced98-26ba-41e4-a7f7-306364f2bbba-istio-data\") pod \"e2e-mt-61280292-openshift-default-59745789f-4wmpz\" (UID: \"903ced98-26ba-41e4-a7f7-306364f2bbba\") " pod="openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz"
Jul 15 21:23:35.802519 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:35.802502    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh9m2\" (UniqueName: \"kubernetes.io/projected/903ced98-26ba-41e4-a7f7-306364f2bbba-kube-api-access-bh9m2\") pod \"e2e-mt-61280292-openshift-default-59745789f-4wmpz\" (UID: \"903ced98-26ba-41e4-a7f7-306364f2bbba\") " pod="openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz"
Jul 15 21:23:35.802664 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:35.802545    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/903ced98-26ba-41e4-a7f7-306364f2bbba-credential-socket\") pod \"e2e-mt-61280292-openshift-default-59745789f-4wmpz\" (UID: \"903ced98-26ba-41e4-a7f7-306364f2bbba\") " pod="openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz"
Jul 15 21:23:35.903874 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:35.903838    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/903ced98-26ba-41e4-a7f7-306364f2bbba-istiod-ca-cert\") pod \"e2e-mt-61280292-openshift-default-59745789f-4wmpz\" (UID: \"903ced98-26ba-41e4-a7f7-306364f2bbba\") " pod="openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz"
Jul 15 21:23:35.904082 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:35.903882    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/903ced98-26ba-41e4-a7f7-306364f2bbba-istio-token\") pod \"e2e-mt-61280292-openshift-default-59745789f-4wmpz\" (UID: \"903ced98-26ba-41e4-a7f7-306364f2bbba\") " pod="openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz"
Jul 15 21:23:35.904082 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:35.903928    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/903ced98-26ba-41e4-a7f7-306364f2bbba-workload-socket\") pod \"e2e-mt-61280292-openshift-default-59745789f-4wmpz\" (UID: \"903ced98-26ba-41e4-a7f7-306364f2bbba\") " pod="openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz"
Jul 15 21:23:35.904082 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:35.903962    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/903ced98-26ba-41e4-a7f7-306364f2bbba-workload-certs\") pod \"e2e-mt-61280292-openshift-default-59745789f-4wmpz\" (UID: \"903ced98-26ba-41e4-a7f7-306364f2bbba\") " pod="openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz"
Jul 15 21:23:35.904082 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:35.903985    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/903ced98-26ba-41e4-a7f7-306364f2bbba-istio-envoy\") pod \"e2e-mt-61280292-openshift-default-59745789f-4wmpz\" (UID: \"903ced98-26ba-41e4-a7f7-306364f2bbba\") " pod="openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz"
Jul 15 21:23:35.904082 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:35.904001    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/903ced98-26ba-41e4-a7f7-306364f2bbba-istio-podinfo\") pod \"e2e-mt-61280292-openshift-default-59745789f-4wmpz\" (UID: \"903ced98-26ba-41e4-a7f7-306364f2bbba\") " pod="openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz"
Jul 15 21:23:35.904082 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:35.904043    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/903ced98-26ba-41e4-a7f7-306364f2bbba-istio-data\") pod \"e2e-mt-61280292-openshift-default-59745789f-4wmpz\" (UID: \"903ced98-26ba-41e4-a7f7-306364f2bbba\") " pod="openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz"
Jul 15 21:23:35.904082 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:35.904067    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh9m2\" (UniqueName: \"kubernetes.io/projected/903ced98-26ba-41e4-a7f7-306364f2bbba-kube-api-access-bh9m2\") pod \"e2e-mt-61280292-openshift-default-59745789f-4wmpz\" (UID: \"903ced98-26ba-41e4-a7f7-306364f2bbba\") " pod="openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz"
Jul 15 21:23:35.904479 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:35.904114    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/903ced98-26ba-41e4-a7f7-306364f2bbba-credential-socket\") pod \"e2e-mt-61280292-openshift-default-59745789f-4wmpz\" (UID: \"903ced98-26ba-41e4-a7f7-306364f2bbba\") " pod="openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz"
Jul 15 21:23:35.904479 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:35.904462    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/903ced98-26ba-41e4-a7f7-306364f2bbba-workload-socket\") pod \"e2e-mt-61280292-openshift-default-59745789f-4wmpz\" (UID: \"903ced98-26ba-41e4-a7f7-306364f2bbba\") " pod="openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz"
Jul 15 21:23:35.904586 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:35.904492    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/903ced98-26ba-41e4-a7f7-306364f2bbba-workload-certs\") pod \"e2e-mt-61280292-openshift-default-59745789f-4wmpz\" (UID: \"903ced98-26ba-41e4-a7f7-306364f2bbba\") " pod="openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz"
Jul 15 21:23:35.904586 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:35.904523    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/903ced98-26ba-41e4-a7f7-306364f2bbba-credential-socket\") pod \"e2e-mt-61280292-openshift-default-59745789f-4wmpz\" (UID: \"903ced98-26ba-41e4-a7f7-306364f2bbba\") " pod="openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz"
Jul 15 21:23:35.904586 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:35.904566    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/903ced98-26ba-41e4-a7f7-306364f2bbba-istio-data\") pod \"e2e-mt-61280292-openshift-default-59745789f-4wmpz\" (UID: \"903ced98-26ba-41e4-a7f7-306364f2bbba\") " pod="openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz"
Jul 15 21:23:35.904737 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:35.904676    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/903ced98-26ba-41e4-a7f7-306364f2bbba-istiod-ca-cert\") pod \"e2e-mt-61280292-openshift-default-59745789f-4wmpz\" (UID: \"903ced98-26ba-41e4-a7f7-306364f2bbba\") " pod="openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz"
Jul 15 21:23:35.906391 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:35.906362    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/903ced98-26ba-41e4-a7f7-306364f2bbba-istio-envoy\") pod \"e2e-mt-61280292-openshift-default-59745789f-4wmpz\" (UID: \"903ced98-26ba-41e4-a7f7-306364f2bbba\") " pod="openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz"
Jul 15 21:23:35.906810 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:35.906789    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/903ced98-26ba-41e4-a7f7-306364f2bbba-istio-podinfo\") pod \"e2e-mt-61280292-openshift-default-59745789f-4wmpz\" (UID: \"903ced98-26ba-41e4-a7f7-306364f2bbba\") " pod="openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz"
Jul 15 21:23:35.915514 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:35.915485    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/903ced98-26ba-41e4-a7f7-306364f2bbba-istio-token\") pod \"e2e-mt-61280292-openshift-default-59745789f-4wmpz\" (UID: \"903ced98-26ba-41e4-a7f7-306364f2bbba\") " pod="openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz"
Jul 15 21:23:35.915979 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:35.915961    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh9m2\" (UniqueName: \"kubernetes.io/projected/903ced98-26ba-41e4-a7f7-306364f2bbba-kube-api-access-bh9m2\") pod \"e2e-mt-61280292-openshift-default-59745789f-4wmpz\" (UID: \"903ced98-26ba-41e4-a7f7-306364f2bbba\") " pod="openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz"
Jul 15 21:23:36.054497 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:36.054399    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz"
Jul 15 21:23:36.198952 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:36.198925    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz"]
Jul 15 21:23:36.201263 ip-10-0-137-234 kubenswrapper[2566]: W0715 21:23:36.201232    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod903ced98_26ba_41e4_a7f7_306364f2bbba.slice/crio-495f6b71b495549b638812d6529481f97b86f561f7e58d81a3af30e903b875cf WatchSource:0}: Error finding container 495f6b71b495549b638812d6529481f97b86f561f7e58d81a3af30e903b875cf: Status 404 returned error can't find the container with id 495f6b71b495549b638812d6529481f97b86f561f7e58d81a3af30e903b875cf
Jul 15 21:23:36.203327 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:36.203297    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:23:36.203390 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:36.203362    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:23:36.203427 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:36.203391    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:23:36.222058 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:36.222033    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:23:36.385342 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:36.385243    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz" event={"ID":"903ced98-26ba-41e4-a7f7-306364f2bbba","Type":"ContainerStarted","Data":"1be9f87cd976ff31abbaa9aa87fb0f15c89e7ddadf45bd37ce1736ee67b72da8"}
Jul 15 21:23:36.385342 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:36.385298    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz" event={"ID":"903ced98-26ba-41e4-a7f7-306364f2bbba","Type":"ContainerStarted","Data":"495f6b71b495549b638812d6529481f97b86f561f7e58d81a3af30e903b875cf"}
Jul 15 21:23:36.424658 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:36.424595    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz" podStartSLOduration=1.424575531 podStartE2EDuration="1.424575531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:23:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:23:36.424002982 +0000 UTC m=+2149.573041038" watchObservedRunningTime="2026-07-15 21:23:36.424575531 +0000 UTC m=+2149.573613591"
Jul 15 21:23:37.055266 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:37.055233    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz"
Jul 15 21:23:37.060281 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:37.060254    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz"
Jul 15 21:23:37.394211 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:37.394122    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz"
Jul 15 21:23:37.395137 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:37.395118    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz"
Jul 15 21:23:38.394505 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:38.394389    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:23:38.483012 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:38.482979    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-61280292-5bcd66bd4d-nrh42"]
Jul 15 21:23:38.483431 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:38.483414    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5f8687c-0a6f-41f1-8c09-4f69e15d3731" containerName="istio-proxy"
Jul 15 21:23:38.483527 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:38.483435    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5f8687c-0a6f-41f1-8c09-4f69e15d3731" containerName="istio-proxy"
Jul 15 21:23:38.486560 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:38.486537    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61280292-5bcd66bd4d-nrh42"
Jul 15 21:23:38.490461 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:38.490432    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:23:38.491880 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:38.491856    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-61280292-serving-cert\""
Jul 15 21:23:38.492257 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:38.492235    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-s2l9d\""
Jul 15 21:23:38.493117 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:38.493096    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 15 21:23:38.493199 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:38.493141    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 15 21:23:38.499349 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:38.499321    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-61280292-5bcd66bd4d-nrh42"]
Jul 15 21:23:38.595473 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:38.595434    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:23:38.628289 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:38.628257    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vprft\" (UniqueName: \"kubernetes.io/projected/126ce765-585e-4229-be2d-ce20a7c978b2-kube-api-access-vprft\") pod \"maas-api-e2e-mt-61280292-5bcd66bd4d-nrh42\" (UID: \"126ce765-585e-4229-be2d-ce20a7c978b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61280292-5bcd66bd4d-nrh42"
Jul 15 21:23:38.628463 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:38.628316    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/126ce765-585e-4229-be2d-ce20a7c978b2-maas-api-tls\") pod \"maas-api-e2e-mt-61280292-5bcd66bd4d-nrh42\" (UID: \"126ce765-585e-4229-be2d-ce20a7c978b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61280292-5bcd66bd4d-nrh42"
Jul 15 21:23:38.628463 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:38.628409    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/126ce765-585e-4229-be2d-ce20a7c978b2-openshift-service-ca\") pod \"maas-api-e2e-mt-61280292-5bcd66bd4d-nrh42\" (UID: \"126ce765-585e-4229-be2d-ce20a7c978b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61280292-5bcd66bd4d-nrh42"
Jul 15 21:23:38.729590 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:38.729555    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/126ce765-585e-4229-be2d-ce20a7c978b2-openshift-service-ca\") pod \"maas-api-e2e-mt-61280292-5bcd66bd4d-nrh42\" (UID: \"126ce765-585e-4229-be2d-ce20a7c978b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61280292-5bcd66bd4d-nrh42"
Jul 15 21:23:38.729770 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:38.729620    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vprft\" (UniqueName: \"kubernetes.io/projected/126ce765-585e-4229-be2d-ce20a7c978b2-kube-api-access-vprft\") pod \"maas-api-e2e-mt-61280292-5bcd66bd4d-nrh42\" (UID: \"126ce765-585e-4229-be2d-ce20a7c978b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61280292-5bcd66bd4d-nrh42"
Jul 15 21:23:38.729770 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:38.729647    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/126ce765-585e-4229-be2d-ce20a7c978b2-maas-api-tls\") pod \"maas-api-e2e-mt-61280292-5bcd66bd4d-nrh42\" (UID: \"126ce765-585e-4229-be2d-ce20a7c978b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61280292-5bcd66bd4d-nrh42"
Jul 15 21:23:38.730247 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:38.730225    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/126ce765-585e-4229-be2d-ce20a7c978b2-openshift-service-ca\") pod \"maas-api-e2e-mt-61280292-5bcd66bd4d-nrh42\" (UID: \"126ce765-585e-4229-be2d-ce20a7c978b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61280292-5bcd66bd4d-nrh42"
Jul 15 21:23:38.732105 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:38.732075    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/126ce765-585e-4229-be2d-ce20a7c978b2-maas-api-tls\") pod \"maas-api-e2e-mt-61280292-5bcd66bd4d-nrh42\" (UID: \"126ce765-585e-4229-be2d-ce20a7c978b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61280292-5bcd66bd4d-nrh42"
Jul 15 21:23:38.740044 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:38.740018    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:23:38.740870 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:38.740848    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vprft\" (UniqueName: \"kubernetes.io/projected/126ce765-585e-4229-be2d-ce20a7c978b2-kube-api-access-vprft\") pod \"maas-api-e2e-mt-61280292-5bcd66bd4d-nrh42\" (UID: \"126ce765-585e-4229-be2d-ce20a7c978b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61280292-5bcd66bd4d-nrh42"
Jul 15 21:23:38.799628 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:38.799596    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61280292-5bcd66bd4d-nrh42"
Jul 15 21:23:38.939116 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:38.939037    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-61280292-5bcd66bd4d-nrh42"]
Jul 15 21:23:38.941369 ip-10-0-137-234 kubenswrapper[2566]: W0715 21:23:38.941333    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod126ce765_585e_4229_be2d_ce20a7c978b2.slice/crio-8022f1f4677f1312f73cdfb683b634772f494d0fa778fc259c687f20d0b372fe WatchSource:0}: Error finding container 8022f1f4677f1312f73cdfb683b634772f494d0fa778fc259c687f20d0b372fe: Status 404 returned error can't find the container with id 8022f1f4677f1312f73cdfb683b634772f494d0fa778fc259c687f20d0b372fe
Jul 15 21:23:39.104148 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:39.104042    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:23:39.205139 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:39.205108    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:23:39.406179 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:39.406140    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61280292-5bcd66bd4d-nrh42" event={"ID":"126ce765-585e-4229-be2d-ce20a7c978b2","Type":"ContainerStarted","Data":"fcda62d85fc8e02fc6795fb46bcb27e38e176a9bf217380a842e6eb6099867e0"}
Jul 15 21:23:39.406549 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:39.406207    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61280292-5bcd66bd4d-nrh42"
Jul 15 21:23:39.406549 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:39.406234    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61280292-5bcd66bd4d-nrh42" event={"ID":"126ce765-585e-4229-be2d-ce20a7c978b2","Type":"ContainerStarted","Data":"8022f1f4677f1312f73cdfb683b634772f494d0fa778fc259c687f20d0b372fe"}
Jul 15 21:23:39.426990 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:39.426942    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61280292-5bcd66bd4d-nrh42" podStartSLOduration=1.066720853 podStartE2EDuration="1.426927788s" totalImagesPullingTime="360.206935ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:23:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:23:39.424790206 +0000 UTC m=+2152.573828264" watchObservedRunningTime="2026-07-15 21:23:39.426927788 +0000 UTC m=+2152.575965841"
Jul 15 21:23:42.922734 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:42.922703    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:23:45.416750 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:45.416720    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61280292-5bcd66bd4d-nrh42"
Jul 15 21:23:48.003684 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:48.003573    2566 scope.go:117] "RemoveContainer" containerID="4f454abb69a30d1c9f3d062751529a4136b915a93521eeb9d0832229b8998ca3"
Jul 15 21:23:52.395456 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:52.395364    2566 patch_prober.go:28] interesting pod/e2e-mt-61280292-openshift-default-59745789f-4wmpz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Jul 15 21:23:52.395456 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:52.395436    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz" podUID="903ced98-26ba-41e4-a7f7-306364f2bbba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Jul 15 21:23:52.472392 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:52.472358    2566 generic.go:356] "Generic (PLEG): container finished" podID="903ced98-26ba-41e4-a7f7-306364f2bbba" containerID="1be9f87cd976ff31abbaa9aa87fb0f15c89e7ddadf45bd37ce1736ee67b72da8" exitCode=0
Jul 15 21:23:52.472536 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:52.472431    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz" event={"ID":"903ced98-26ba-41e4-a7f7-306364f2bbba","Type":"ContainerDied","Data":"1be9f87cd976ff31abbaa9aa87fb0f15c89e7ddadf45bd37ce1736ee67b72da8"}
Jul 15 21:23:52.472936 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:52.472916    2566 scope.go:117] "RemoveContainer" containerID="1be9f87cd976ff31abbaa9aa87fb0f15c89e7ddadf45bd37ce1736ee67b72da8"
Jul 15 21:23:52.473966 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:52.473932    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:23:52.474088 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:52.474024    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:23:52.474088 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:52.474075    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:23:53.481978 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:53.481935    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz" event={"ID":"903ced98-26ba-41e4-a7f7-306364f2bbba","Type":"ContainerStarted","Data":"e1d5cc38e2896f748d4c684f869f1623bfc658f7f1690152c00d39355873a7cc"}
Jul 15 21:23:54.055227 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:54.055193    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz"
Jul 15 21:23:54.055407 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:54.055371    2566 patch_prober.go:28] interesting pod/e2e-mt-61280292-openshift-default-59745789f-4wmpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Jul 15 21:23:54.055453 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:54.055418    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz" podUID="903ced98-26ba-41e4-a7f7-306364f2bbba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Jul 15 21:23:55.055186 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:55.055150    2566 patch_prober.go:28] interesting pod/e2e-mt-61280292-openshift-default-59745789f-4wmpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Jul 15 21:23:55.055605 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:55.055216    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz" podUID="903ced98-26ba-41e4-a7f7-306364f2bbba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Jul 15 21:23:56.054974 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:56.054936    2566 patch_prober.go:28] interesting pod/e2e-mt-61280292-openshift-default-59745789f-4wmpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Jul 15 21:23:56.055162 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:56.054995    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz" podUID="903ced98-26ba-41e4-a7f7-306364f2bbba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Jul 15 21:23:57.054974 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:57.054943    2566 patch_prober.go:28] interesting pod/e2e-mt-61280292-openshift-default-59745789f-4wmpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Jul 15 21:23:57.055340 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:57.055005    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz" podUID="903ced98-26ba-41e4-a7f7-306364f2bbba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Jul 15 21:23:58.058452 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:58.058424    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz"
Jul 15 21:23:58.058846 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:58.058682    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz"
Jul 15 21:23:58.059482 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:23:58.059466    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz"
Jul 15 21:24:08.667097 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:24:08.667055    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:24:11.428986 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:24:11.428946    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:24:11.529487 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:24:11.529457    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:24:12.176164 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:24:12.176117    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:24:36.900028 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:24:36.899995    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61280292-tphkg"]
Jul 15 21:24:36.903420 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:24:36.903397    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61280292-tphkg"
Jul 15 21:24:36.906761 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:24:36.906742    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zsrf5\""
Jul 15 21:24:36.912815 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:24:36.912784    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61280292-tphkg"]
Jul 15 21:24:37.065648 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:24:37.065615    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92c9974e-5ce2-40fe-aa2b-15994743a444-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-61280292-tphkg\" (UID: \"92c9974e-5ce2-40fe-aa2b-15994743a444\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61280292-tphkg"
Jul 15 21:24:37.167110 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:24:37.167014    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92c9974e-5ce2-40fe-aa2b-15994743a444-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-61280292-tphkg\" (UID: \"92c9974e-5ce2-40fe-aa2b-15994743a444\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61280292-tphkg"
Jul 15 21:24:37.167646 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:24:37.167625    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92c9974e-5ce2-40fe-aa2b-15994743a444-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-61280292-tphkg\" (UID: \"92c9974e-5ce2-40fe-aa2b-15994743a444\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61280292-tphkg"
Jul 15 21:24:37.215270 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:24:37.215234    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61280292-tphkg"
Jul 15 21:24:37.359294 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:24:37.359263    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61280292-tphkg"]
Jul 15 21:24:37.361080 ip-10-0-137-234 kubenswrapper[2566]: W0715 21:24:37.361047    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92c9974e_5ce2_40fe_aa2b_15994743a444.slice/crio-92155ae07f137341e8e8cb76270f1d8670556d8599e9fb7427e44cb903c7a91c WatchSource:0}: Error finding container 92155ae07f137341e8e8cb76270f1d8670556d8599e9fb7427e44cb903c7a91c: Status 404 returned error can't find the container with id 92155ae07f137341e8e8cb76270f1d8670556d8599e9fb7427e44cb903c7a91c
Jul 15 21:24:37.704988 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:24:37.704952    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61280292-tphkg" event={"ID":"92c9974e-5ce2-40fe-aa2b-15994743a444","Type":"ContainerStarted","Data":"4e3ab711d27ab001e7ac4b053f46818ff8c452d04e7e072194521115076188f3"}
Jul 15 21:24:37.705152 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:24:37.705014    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61280292-tphkg" event={"ID":"92c9974e-5ce2-40fe-aa2b-15994743a444","Type":"ContainerStarted","Data":"92155ae07f137341e8e8cb76270f1d8670556d8599e9fb7427e44cb903c7a91c"}
Jul 15 21:24:37.724425 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:24:37.724370    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61280292-tphkg" podStartSLOduration=1.7243543959999998 podStartE2EDuration="1.724354396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:24:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:24:37.722507606 +0000 UTC m=+2210.871545664" watchObservedRunningTime="2026-07-15 21:24:37.724354396 +0000 UTC m=+2210.873392453"
Jul 15 21:24:38.711077 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:24:38.711041    2566 generic.go:356] "Generic (PLEG): container finished" podID="92c9974e-5ce2-40fe-aa2b-15994743a444" containerID="4e3ab711d27ab001e7ac4b053f46818ff8c452d04e7e072194521115076188f3" exitCode=0
Jul 15 21:24:38.711454 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:24:38.711107    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61280292-tphkg" event={"ID":"92c9974e-5ce2-40fe-aa2b-15994743a444","Type":"ContainerDied","Data":"4e3ab711d27ab001e7ac4b053f46818ff8c452d04e7e072194521115076188f3"}
Jul 15 21:24:39.842332 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:24:39.842306    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61280292-tphkg"
Jul 15 21:24:39.988045 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:24:39.987960    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92c9974e-5ce2-40fe-aa2b-15994743a444-maas-api-service-ca\") pod \"92c9974e-5ce2-40fe-aa2b-15994743a444\" (UID: \"92c9974e-5ce2-40fe-aa2b-15994743a444\") "
Jul 15 21:24:39.988284 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:24:39.988263    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92c9974e-5ce2-40fe-aa2b-15994743a444-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "92c9974e-5ce2-40fe-aa2b-15994743a444" (UID: "92c9974e-5ce2-40fe-aa2b-15994743a444"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 21:24:40.088797 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:24:40.088759    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92c9974e-5ce2-40fe-aa2b-15994743a444-maas-api-service-ca\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:24:40.721316 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:24:40.721281    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61280292-tphkg" event={"ID":"92c9974e-5ce2-40fe-aa2b-15994743a444","Type":"ContainerDied","Data":"92155ae07f137341e8e8cb76270f1d8670556d8599e9fb7427e44cb903c7a91c"}
Jul 15 21:24:40.721524 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:24:40.721322    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61280292-tphkg"
Jul 15 21:24:40.721524 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:24:40.721330    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92155ae07f137341e8e8cb76270f1d8670556d8599e9fb7427e44cb903c7a91c"
Jul 15 21:24:47.880186 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:24:47.880152    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-61280292-5bcd66bd4d-nrh42"]
Jul 15 21:24:47.880631 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:24:47.880358    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61280292-5bcd66bd4d-nrh42" podUID="126ce765-585e-4229-be2d-ce20a7c978b2" containerName="maas-api" containerID="cri-o://fcda62d85fc8e02fc6795fb46bcb27e38e176a9bf217380a842e6eb6099867e0" gracePeriod=30
Jul 15 21:24:47.912083 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:24:47.912047    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:24:48.037805 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:24:48.037774    2566 scope.go:117] "RemoveContainer" containerID="e63a87ffe99163aaf02d496ace3dc3f9f9b349c8078f10e6d9a1f9a56e2095a1"
Jul 15 21:24:48.141673 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:24:48.141605    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61280292-5bcd66bd4d-nrh42"
Jul 15 21:24:48.154718 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:24:48.154692    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vprft\" (UniqueName: \"kubernetes.io/projected/126ce765-585e-4229-be2d-ce20a7c978b2-kube-api-access-vprft\") pod \"126ce765-585e-4229-be2d-ce20a7c978b2\" (UID: \"126ce765-585e-4229-be2d-ce20a7c978b2\") "
Jul 15 21:24:48.154856 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:24:48.154725    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/126ce765-585e-4229-be2d-ce20a7c978b2-maas-api-tls\") pod \"126ce765-585e-4229-be2d-ce20a7c978b2\" (UID: \"126ce765-585e-4229-be2d-ce20a7c978b2\") "
Jul 15 21:24:48.154856 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:24:48.154781    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/126ce765-585e-4229-be2d-ce20a7c978b2-openshift-service-ca\") pod \"126ce765-585e-4229-be2d-ce20a7c978b2\" (UID: \"126ce765-585e-4229-be2d-ce20a7c978b2\") "
Jul 15 21:24:48.155185 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:24:48.155160    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/126ce765-585e-4229-be2d-ce20a7c978b2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "126ce765-585e-4229-be2d-ce20a7c978b2" (UID: "126ce765-585e-4229-be2d-ce20a7c978b2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 21:24:48.156869 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:24:48.156842    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/126ce765-585e-4229-be2d-ce20a7c978b2-kube-api-access-vprft" (OuterVolumeSpecName: "kube-api-access-vprft") pod "126ce765-585e-4229-be2d-ce20a7c978b2" (UID: "126ce765-585e-4229-be2d-ce20a7c978b2"). InnerVolumeSpecName "kube-api-access-vprft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 21:24:48.156996 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:24:48.156845    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/126ce765-585e-4229-be2d-ce20a7c978b2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "126ce765-585e-4229-be2d-ce20a7c978b2" (UID: "126ce765-585e-4229-be2d-ce20a7c978b2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 21:24:48.255705 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:24:48.255674    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vprft\" (UniqueName: \"kubernetes.io/projected/126ce765-585e-4229-be2d-ce20a7c978b2-kube-api-access-vprft\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:24:48.255705 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:24:48.255700    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/126ce765-585e-4229-be2d-ce20a7c978b2-maas-api-tls\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:24:48.255705 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:24:48.255710    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/126ce765-585e-4229-be2d-ce20a7c978b2-openshift-service-ca\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:24:48.759989 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:24:48.759953    2566 generic.go:356] "Generic (PLEG): container finished" podID="126ce765-585e-4229-be2d-ce20a7c978b2" containerID="fcda62d85fc8e02fc6795fb46bcb27e38e176a9bf217380a842e6eb6099867e0" exitCode=0
Jul 15 21:24:48.760164 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:24:48.760001    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61280292-5bcd66bd4d-nrh42"
Jul 15 21:24:48.760164 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:24:48.760038    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61280292-5bcd66bd4d-nrh42" event={"ID":"126ce765-585e-4229-be2d-ce20a7c978b2","Type":"ContainerDied","Data":"fcda62d85fc8e02fc6795fb46bcb27e38e176a9bf217380a842e6eb6099867e0"}
Jul 15 21:24:48.760164 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:24:48.760094    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61280292-5bcd66bd4d-nrh42" event={"ID":"126ce765-585e-4229-be2d-ce20a7c978b2","Type":"ContainerDied","Data":"8022f1f4677f1312f73cdfb683b634772f494d0fa778fc259c687f20d0b372fe"}
Jul 15 21:24:48.760164 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:24:48.760125    2566 scope.go:117] "RemoveContainer" containerID="fcda62d85fc8e02fc6795fb46bcb27e38e176a9bf217380a842e6eb6099867e0"
Jul 15 21:24:48.769693 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:24:48.769677    2566 scope.go:117] "RemoveContainer" containerID="fcda62d85fc8e02fc6795fb46bcb27e38e176a9bf217380a842e6eb6099867e0"
Jul 15 21:24:48.769957 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:24:48.769938    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcda62d85fc8e02fc6795fb46bcb27e38e176a9bf217380a842e6eb6099867e0\": container with ID starting with fcda62d85fc8e02fc6795fb46bcb27e38e176a9bf217380a842e6eb6099867e0 not found: ID does not exist" containerID="fcda62d85fc8e02fc6795fb46bcb27e38e176a9bf217380a842e6eb6099867e0"
Jul 15 21:24:48.770013 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:24:48.769966    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcda62d85fc8e02fc6795fb46bcb27e38e176a9bf217380a842e6eb6099867e0"} err="failed to get container status \"fcda62d85fc8e02fc6795fb46bcb27e38e176a9bf217380a842e6eb6099867e0\": rpc error: code = NotFound desc = could not find container \"fcda62d85fc8e02fc6795fb46bcb27e38e176a9bf217380a842e6eb6099867e0\": container with ID starting with fcda62d85fc8e02fc6795fb46bcb27e38e176a9bf217380a842e6eb6099867e0 not found: ID does not exist"
Jul 15 21:24:48.783761 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:24:48.783730    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-61280292-5bcd66bd4d-nrh42"]
Jul 15 21:24:48.789443 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:24:48.789417    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-61280292-5bcd66bd4d-nrh42"]
Jul 15 21:24:49.756797 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:24:49.756766    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="126ce765-585e-4229-be2d-ce20a7c978b2" path="/var/lib/kubelet/pods/126ce765-585e-4229-be2d-ce20a7c978b2/volumes"
Jul 15 21:24:52.003696 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:24:52.003656    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:24:57.070884 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:24:57.070840    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:25:33.562376 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:33.562283    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:25:35.000105 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:35.000069    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:25:41.033828 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:41.033792    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:25:41.145552 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:41.145521    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:25:44.443491 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:44.443447    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz"]
Jul 15 21:25:44.443960 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:44.443693    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz" podUID="903ced98-26ba-41e4-a7f7-306364f2bbba" containerName="istio-proxy" containerID="cri-o://e1d5cc38e2896f748d4c684f869f1623bfc658f7f1690152c00d39355873a7cc" gracePeriod=30
Jul 15 21:25:44.461987 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:44.461943    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:25:44.706548 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:44.706452    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:25:45.856757 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:45.856708    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:25:45.948584 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:45.948549    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:25:46.049484 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:46.049448    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:25:46.908833 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:46.908802    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:25:47.010121 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:47.010082    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:25:47.109997 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:47.109944    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:25:48.062116 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:48.062082    2566 scope.go:117] "RemoveContainer" containerID="0d5c83eb57852f2d77d3e1809f4d671819ec61bf7a955f5e03fd953ade4e95d8"
Jul 15 21:25:48.650267 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:48.650201    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:25:49.705256 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:49.705192    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz"
Jul 15 21:25:49.788065 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:49.788030    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bh9m2\" (UniqueName: \"kubernetes.io/projected/903ced98-26ba-41e4-a7f7-306364f2bbba-kube-api-access-bh9m2\") pod \"903ced98-26ba-41e4-a7f7-306364f2bbba\" (UID: \"903ced98-26ba-41e4-a7f7-306364f2bbba\") "
Jul 15 21:25:49.788065 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:49.788069    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/903ced98-26ba-41e4-a7f7-306364f2bbba-istiod-ca-cert\") pod \"903ced98-26ba-41e4-a7f7-306364f2bbba\" (UID: \"903ced98-26ba-41e4-a7f7-306364f2bbba\") "
Jul 15 21:25:49.788306 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:49.788092    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/903ced98-26ba-41e4-a7f7-306364f2bbba-istio-envoy\") pod \"903ced98-26ba-41e4-a7f7-306364f2bbba\" (UID: \"903ced98-26ba-41e4-a7f7-306364f2bbba\") "
Jul 15 21:25:49.788306 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:49.788225    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/903ced98-26ba-41e4-a7f7-306364f2bbba-workload-certs\") pod \"903ced98-26ba-41e4-a7f7-306364f2bbba\" (UID: \"903ced98-26ba-41e4-a7f7-306364f2bbba\") "
Jul 15 21:25:49.788306 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:49.788268    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/903ced98-26ba-41e4-a7f7-306364f2bbba-credential-socket\") pod \"903ced98-26ba-41e4-a7f7-306364f2bbba\" (UID: \"903ced98-26ba-41e4-a7f7-306364f2bbba\") "
Jul 15 21:25:49.788470 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:49.788313    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/903ced98-26ba-41e4-a7f7-306364f2bbba-workload-socket\") pod \"903ced98-26ba-41e4-a7f7-306364f2bbba\" (UID: \"903ced98-26ba-41e4-a7f7-306364f2bbba\") "
Jul 15 21:25:49.788470 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:49.788344    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/903ced98-26ba-41e4-a7f7-306364f2bbba-istio-data\") pod \"903ced98-26ba-41e4-a7f7-306364f2bbba\" (UID: \"903ced98-26ba-41e4-a7f7-306364f2bbba\") "
Jul 15 21:25:49.788470 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:49.788373    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/903ced98-26ba-41e4-a7f7-306364f2bbba-istio-token\") pod \"903ced98-26ba-41e4-a7f7-306364f2bbba\" (UID: \"903ced98-26ba-41e4-a7f7-306364f2bbba\") "
Jul 15 21:25:49.788470 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:49.788429    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/903ced98-26ba-41e4-a7f7-306364f2bbba-istio-podinfo\") pod \"903ced98-26ba-41e4-a7f7-306364f2bbba\" (UID: \"903ced98-26ba-41e4-a7f7-306364f2bbba\") "
Jul 15 21:25:49.788652 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:49.788486    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/903ced98-26ba-41e4-a7f7-306364f2bbba-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "903ced98-26ba-41e4-a7f7-306364f2bbba" (UID: "903ced98-26ba-41e4-a7f7-306364f2bbba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 21:25:49.788652 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:49.788579    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/903ced98-26ba-41e4-a7f7-306364f2bbba-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "903ced98-26ba-41e4-a7f7-306364f2bbba" (UID: "903ced98-26ba-41e4-a7f7-306364f2bbba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:25:49.788652 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:49.788616    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/903ced98-26ba-41e4-a7f7-306364f2bbba-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "903ced98-26ba-41e4-a7f7-306364f2bbba" (UID: "903ced98-26ba-41e4-a7f7-306364f2bbba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:25:49.788821 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:49.788751    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/903ced98-26ba-41e4-a7f7-306364f2bbba-credential-socket\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:25:49.788821 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:49.788771    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/903ced98-26ba-41e4-a7f7-306364f2bbba-workload-socket\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:25:49.788821 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:49.788787    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/903ced98-26ba-41e4-a7f7-306364f2bbba-istiod-ca-cert\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:25:49.788821 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:49.788783    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/903ced98-26ba-41e4-a7f7-306364f2bbba-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "903ced98-26ba-41e4-a7f7-306364f2bbba" (UID: "903ced98-26ba-41e4-a7f7-306364f2bbba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:25:49.789364 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:49.789333    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/903ced98-26ba-41e4-a7f7-306364f2bbba-istio-data" (OuterVolumeSpecName: "istio-data") pod "903ced98-26ba-41e4-a7f7-306364f2bbba" (UID: "903ced98-26ba-41e4-a7f7-306364f2bbba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:25:49.790360 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:49.790333    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/903ced98-26ba-41e4-a7f7-306364f2bbba-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "903ced98-26ba-41e4-a7f7-306364f2bbba" (UID: "903ced98-26ba-41e4-a7f7-306364f2bbba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:25:49.790648 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:49.790627    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/903ced98-26ba-41e4-a7f7-306364f2bbba-kube-api-access-bh9m2" (OuterVolumeSpecName: "kube-api-access-bh9m2") pod "903ced98-26ba-41e4-a7f7-306364f2bbba" (UID: "903ced98-26ba-41e4-a7f7-306364f2bbba"). InnerVolumeSpecName "kube-api-access-bh9m2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 21:25:49.790723 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:49.790691    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/903ced98-26ba-41e4-a7f7-306364f2bbba-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "903ced98-26ba-41e4-a7f7-306364f2bbba" (UID: "903ced98-26ba-41e4-a7f7-306364f2bbba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 15 21:25:49.790821 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:49.790806    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/903ced98-26ba-41e4-a7f7-306364f2bbba-istio-token" (OuterVolumeSpecName: "istio-token") pod "903ced98-26ba-41e4-a7f7-306364f2bbba" (UID: "903ced98-26ba-41e4-a7f7-306364f2bbba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 21:25:49.889275 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:49.889244    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/903ced98-26ba-41e4-a7f7-306364f2bbba-istio-data\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:25:49.889275 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:49.889273    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/903ced98-26ba-41e4-a7f7-306364f2bbba-istio-token\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:25:49.889275 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:49.889282    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/903ced98-26ba-41e4-a7f7-306364f2bbba-istio-podinfo\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:25:49.889500 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:49.889292    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh9m2\" (UniqueName: \"kubernetes.io/projected/903ced98-26ba-41e4-a7f7-306364f2bbba-kube-api-access-bh9m2\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:25:49.889500 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:49.889302    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/903ced98-26ba-41e4-a7f7-306364f2bbba-istio-envoy\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:25:49.889500 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:49.889310    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/903ced98-26ba-41e4-a7f7-306364f2bbba-workload-certs\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:25:50.053875 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:50.053779    2566 generic.go:356] "Generic (PLEG): container finished" podID="903ced98-26ba-41e4-a7f7-306364f2bbba" containerID="e1d5cc38e2896f748d4c684f869f1623bfc658f7f1690152c00d39355873a7cc" exitCode=0
Jul 15 21:25:50.053875 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:50.053832    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz"
Jul 15 21:25:50.053875 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:50.053859    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz" event={"ID":"903ced98-26ba-41e4-a7f7-306364f2bbba","Type":"ContainerDied","Data":"e1d5cc38e2896f748d4c684f869f1623bfc658f7f1690152c00d39355873a7cc"}
Jul 15 21:25:50.054184 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:50.053935    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz" event={"ID":"903ced98-26ba-41e4-a7f7-306364f2bbba","Type":"ContainerDied","Data":"495f6b71b495549b638812d6529481f97b86f561f7e58d81a3af30e903b875cf"}
Jul 15 21:25:50.054184 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:50.053959    2566 scope.go:117] "RemoveContainer" containerID="e1d5cc38e2896f748d4c684f869f1623bfc658f7f1690152c00d39355873a7cc"
Jul 15 21:25:50.063350 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:50.063331    2566 scope.go:117] "RemoveContainer" containerID="1be9f87cd976ff31abbaa9aa87fb0f15c89e7ddadf45bd37ce1736ee67b72da8"
Jul 15 21:25:50.071569 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:50.071551    2566 scope.go:117] "RemoveContainer" containerID="e1d5cc38e2896f748d4c684f869f1623bfc658f7f1690152c00d39355873a7cc"
Jul 15 21:25:50.071782 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:25:50.071767    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1d5cc38e2896f748d4c684f869f1623bfc658f7f1690152c00d39355873a7cc\": container with ID starting with e1d5cc38e2896f748d4c684f869f1623bfc658f7f1690152c00d39355873a7cc not found: ID does not exist" containerID="e1d5cc38e2896f748d4c684f869f1623bfc658f7f1690152c00d39355873a7cc"
Jul 15 21:25:50.071836 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:50.071789    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1d5cc38e2896f748d4c684f869f1623bfc658f7f1690152c00d39355873a7cc"} err="failed to get container status \"e1d5cc38e2896f748d4c684f869f1623bfc658f7f1690152c00d39355873a7cc\": rpc error: code = NotFound desc = could not find container \"e1d5cc38e2896f748d4c684f869f1623bfc658f7f1690152c00d39355873a7cc\": container with ID starting with e1d5cc38e2896f748d4c684f869f1623bfc658f7f1690152c00d39355873a7cc not found: ID does not exist"
Jul 15 21:25:50.071836 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:50.071805    2566 scope.go:117] "RemoveContainer" containerID="1be9f87cd976ff31abbaa9aa87fb0f15c89e7ddadf45bd37ce1736ee67b72da8"
Jul 15 21:25:50.072049 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:25:50.072029    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1be9f87cd976ff31abbaa9aa87fb0f15c89e7ddadf45bd37ce1736ee67b72da8\": container with ID starting with 1be9f87cd976ff31abbaa9aa87fb0f15c89e7ddadf45bd37ce1736ee67b72da8 not found: ID does not exist" containerID="1be9f87cd976ff31abbaa9aa87fb0f15c89e7ddadf45bd37ce1736ee67b72da8"
Jul 15 21:25:50.072102 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:50.072055    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1be9f87cd976ff31abbaa9aa87fb0f15c89e7ddadf45bd37ce1736ee67b72da8"} err="failed to get container status \"1be9f87cd976ff31abbaa9aa87fb0f15c89e7ddadf45bd37ce1736ee67b72da8\": rpc error: code = NotFound desc = could not find container \"1be9f87cd976ff31abbaa9aa87fb0f15c89e7ddadf45bd37ce1736ee67b72da8\": container with ID starting with 1be9f87cd976ff31abbaa9aa87fb0f15c89e7ddadf45bd37ce1736ee67b72da8 not found: ID does not exist"
Jul 15 21:25:50.081076 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:50.081050    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz"]
Jul 15 21:25:50.082829 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:50.082807    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-61280292-openshift-default-59745789f-4wmpz"]
Jul 15 21:25:51.755962 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:51.755932    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="903ced98-26ba-41e4-a7f7-306364f2bbba" path="/var/lib/kubelet/pods/903ced98-26ba-41e4-a7f7-306364f2bbba/volumes"
Jul 15 21:25:55.112969 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:55.112922    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:25:55.208869 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:55.208832    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:25:55.847504 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:25:55.847454    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:26:02.510052 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:02.510013    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:26:02.585306 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:02.585253    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:26:02.680960 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:02.680924    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:26:02.776677 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:02.776582    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:26:13.568245 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:13.568207    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:26:13.705687 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:13.705647    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:26:34.902702 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:34.902662    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:26:41.382326 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:41.382233    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:26:41.464888 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:41.464857    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:26:41.668527 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:41.668440    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:26:41.996571 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:41.996540    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r"]
Jul 15 21:26:41.996922 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:41.996887    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="126ce765-585e-4229-be2d-ce20a7c978b2" containerName="maas-api"
Jul 15 21:26:41.996922 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:41.996915    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="126ce765-585e-4229-be2d-ce20a7c978b2" containerName="maas-api"
Jul 15 21:26:41.996922 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:41.996924    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92c9974e-5ce2-40fe-aa2b-15994743a444" containerName="revoke-keys"
Jul 15 21:26:41.997162 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:41.996930    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="92c9974e-5ce2-40fe-aa2b-15994743a444" containerName="revoke-keys"
Jul 15 21:26:41.997162 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:41.996945    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="903ced98-26ba-41e4-a7f7-306364f2bbba" containerName="istio-proxy"
Jul 15 21:26:41.997162 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:41.996952    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="903ced98-26ba-41e4-a7f7-306364f2bbba" containerName="istio-proxy"
Jul 15 21:26:41.997162 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:41.997027    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="92c9974e-5ce2-40fe-aa2b-15994743a444" containerName="revoke-keys"
Jul 15 21:26:41.997162 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:41.997036    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="126ce765-585e-4229-be2d-ce20a7c978b2" containerName="maas-api"
Jul 15 21:26:41.997162 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:41.997044    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="903ced98-26ba-41e4-a7f7-306364f2bbba" containerName="istio-proxy"
Jul 15 21:26:42.000217 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:42.000191    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r"
Jul 15 21:26:42.003342 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:42.003314    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 15 21:26:42.003484 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:42.003391    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d87a1e-openshift-default-dockercfg-5xvns\""
Jul 15 21:26:42.015504 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:42.015478    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r"]
Jul 15 21:26:42.032601 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:42.032524    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:26:42.142518 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:42.142477    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2ppn\" (UniqueName: \"kubernetes.io/projected/fd207757-2b22-4a73-838e-e0ed514ee510-kube-api-access-v2ppn\") pod \"e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r\" (UID: \"fd207757-2b22-4a73-838e-e0ed514ee510\") " pod="openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r"
Jul 15 21:26:42.142698 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:42.142547    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd207757-2b22-4a73-838e-e0ed514ee510-istio-podinfo\") pod \"e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r\" (UID: \"fd207757-2b22-4a73-838e-e0ed514ee510\") " pod="openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r"
Jul 15 21:26:42.142698 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:42.142614    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd207757-2b22-4a73-838e-e0ed514ee510-workload-certs\") pod \"e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r\" (UID: \"fd207757-2b22-4a73-838e-e0ed514ee510\") " pod="openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r"
Jul 15 21:26:42.142698 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:42.142682    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd207757-2b22-4a73-838e-e0ed514ee510-workload-socket\") pod \"e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r\" (UID: \"fd207757-2b22-4a73-838e-e0ed514ee510\") " pod="openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r"
Jul 15 21:26:42.142815 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:42.142712    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd207757-2b22-4a73-838e-e0ed514ee510-istio-token\") pod \"e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r\" (UID: \"fd207757-2b22-4a73-838e-e0ed514ee510\") " pod="openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r"
Jul 15 21:26:42.142815 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:42.142753    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd207757-2b22-4a73-838e-e0ed514ee510-istio-envoy\") pod \"e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r\" (UID: \"fd207757-2b22-4a73-838e-e0ed514ee510\") " pod="openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r"
Jul 15 21:26:42.142815 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:42.142771    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd207757-2b22-4a73-838e-e0ed514ee510-istio-data\") pod \"e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r\" (UID: \"fd207757-2b22-4a73-838e-e0ed514ee510\") " pod="openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r"
Jul 15 21:26:42.142815 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:42.142801    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd207757-2b22-4a73-838e-e0ed514ee510-credential-socket\") pod \"e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r\" (UID: \"fd207757-2b22-4a73-838e-e0ed514ee510\") " pod="openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r"
Jul 15 21:26:42.142984 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:42.142858    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd207757-2b22-4a73-838e-e0ed514ee510-istiod-ca-cert\") pod \"e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r\" (UID: \"fd207757-2b22-4a73-838e-e0ed514ee510\") " pod="openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r"
Jul 15 21:26:42.243862 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:42.243828    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2ppn\" (UniqueName: \"kubernetes.io/projected/fd207757-2b22-4a73-838e-e0ed514ee510-kube-api-access-v2ppn\") pod \"e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r\" (UID: \"fd207757-2b22-4a73-838e-e0ed514ee510\") " pod="openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r"
Jul 15 21:26:42.244089 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:42.243872    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd207757-2b22-4a73-838e-e0ed514ee510-istio-podinfo\") pod \"e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r\" (UID: \"fd207757-2b22-4a73-838e-e0ed514ee510\") " pod="openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r"
Jul 15 21:26:42.244089 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:42.243961    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd207757-2b22-4a73-838e-e0ed514ee510-workload-certs\") pod \"e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r\" (UID: \"fd207757-2b22-4a73-838e-e0ed514ee510\") " pod="openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r"
Jul 15 21:26:42.244089 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:42.244015    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd207757-2b22-4a73-838e-e0ed514ee510-workload-socket\") pod \"e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r\" (UID: \"fd207757-2b22-4a73-838e-e0ed514ee510\") " pod="openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r"
Jul 15 21:26:42.244089 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:42.244039    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd207757-2b22-4a73-838e-e0ed514ee510-istio-token\") pod \"e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r\" (UID: \"fd207757-2b22-4a73-838e-e0ed514ee510\") " pod="openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r"
Jul 15 21:26:42.244311 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:42.244096    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd207757-2b22-4a73-838e-e0ed514ee510-istio-envoy\") pod \"e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r\" (UID: \"fd207757-2b22-4a73-838e-e0ed514ee510\") " pod="openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r"
Jul 15 21:26:42.244311 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:42.244122    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd207757-2b22-4a73-838e-e0ed514ee510-istio-data\") pod \"e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r\" (UID: \"fd207757-2b22-4a73-838e-e0ed514ee510\") " pod="openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r"
Jul 15 21:26:42.244311 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:42.244176    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd207757-2b22-4a73-838e-e0ed514ee510-credential-socket\") pod \"e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r\" (UID: \"fd207757-2b22-4a73-838e-e0ed514ee510\") " pod="openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r"
Jul 15 21:26:42.244471 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:42.244440    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd207757-2b22-4a73-838e-e0ed514ee510-istiod-ca-cert\") pod \"e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r\" (UID: \"fd207757-2b22-4a73-838e-e0ed514ee510\") " pod="openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r"
Jul 15 21:26:42.244471 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:42.244457    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd207757-2b22-4a73-838e-e0ed514ee510-workload-certs\") pod \"e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r\" (UID: \"fd207757-2b22-4a73-838e-e0ed514ee510\") " pod="openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r"
Jul 15 21:26:42.244567 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:42.244482    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd207757-2b22-4a73-838e-e0ed514ee510-workload-socket\") pod \"e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r\" (UID: \"fd207757-2b22-4a73-838e-e0ed514ee510\") " pod="openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r"
Jul 15 21:26:42.244567 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:42.244555    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd207757-2b22-4a73-838e-e0ed514ee510-credential-socket\") pod \"e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r\" (UID: \"fd207757-2b22-4a73-838e-e0ed514ee510\") " pod="openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r"
Jul 15 21:26:42.245024 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:42.244622    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd207757-2b22-4a73-838e-e0ed514ee510-istio-data\") pod \"e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r\" (UID: \"fd207757-2b22-4a73-838e-e0ed514ee510\") " pod="openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r"
Jul 15 21:26:42.245144 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:42.245124    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd207757-2b22-4a73-838e-e0ed514ee510-istiod-ca-cert\") pod \"e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r\" (UID: \"fd207757-2b22-4a73-838e-e0ed514ee510\") " pod="openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r"
Jul 15 21:26:42.246458 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:42.246426    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd207757-2b22-4a73-838e-e0ed514ee510-istio-envoy\") pod \"e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r\" (UID: \"fd207757-2b22-4a73-838e-e0ed514ee510\") " pod="openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r"
Jul 15 21:26:42.246565 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:42.246542    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd207757-2b22-4a73-838e-e0ed514ee510-istio-podinfo\") pod \"e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r\" (UID: \"fd207757-2b22-4a73-838e-e0ed514ee510\") " pod="openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r"
Jul 15 21:26:42.266516 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:42.266450    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd207757-2b22-4a73-838e-e0ed514ee510-istio-token\") pod \"e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r\" (UID: \"fd207757-2b22-4a73-838e-e0ed514ee510\") " pod="openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r"
Jul 15 21:26:42.271811 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:42.271792    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2ppn\" (UniqueName: \"kubernetes.io/projected/fd207757-2b22-4a73-838e-e0ed514ee510-kube-api-access-v2ppn\") pod \"e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r\" (UID: \"fd207757-2b22-4a73-838e-e0ed514ee510\") " pod="openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r"
Jul 15 21:26:42.289526 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:42.289498    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:26:42.312538 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:42.312510    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r"
Jul 15 21:26:42.454925 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:42.454888    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r"]
Jul 15 21:26:42.457062 ip-10-0-137-234 kubenswrapper[2566]: W0715 21:26:42.457022    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd207757_2b22_4a73_838e_e0ed514ee510.slice/crio-a27e1455f133e7cdc4b58c15a97ea5606e2bfb46024ab7f5eb6a155be0b15365 WatchSource:0}: Error finding container a27e1455f133e7cdc4b58c15a97ea5606e2bfb46024ab7f5eb6a155be0b15365: Status 404 returned error can't find the container with id a27e1455f133e7cdc4b58c15a97ea5606e2bfb46024ab7f5eb6a155be0b15365
Jul 15 21:26:42.459073 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:42.459041    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:26:42.459160 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:42.459105    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:26:42.459160 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:42.459135    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:26:43.302175 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:43.302138    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r" event={"ID":"fd207757-2b22-4a73-838e-e0ed514ee510","Type":"ContainerStarted","Data":"1ad2933093fb1225e688e1a79dfabd378b37372deb9d27aba934d92c8807e5bc"}
Jul 15 21:26:43.302372 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:43.302200    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r" event={"ID":"fd207757-2b22-4a73-838e-e0ed514ee510","Type":"ContainerStarted","Data":"a27e1455f133e7cdc4b58c15a97ea5606e2bfb46024ab7f5eb6a155be0b15365"}
Jul 15 21:26:43.313376 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:43.313349    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r"
Jul 15 21:26:43.318108 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:43.318084    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r"
Jul 15 21:26:43.326402 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:43.326349    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r" podStartSLOduration=2.326331614 podStartE2EDuration="2.326331614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:26:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:26:43.325588358 +0000 UTC m=+2336.474626434" watchObservedRunningTime="2026-07-15 21:26:43.326331614 +0000 UTC m=+2336.475369671"
Jul 15 21:26:44.307258 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:44.307225    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r"
Jul 15 21:26:44.308312 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:44.308293    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r"
Jul 15 21:26:45.431128 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:45.431095    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d87a1e-7b4b74bc96-p2gdn"]
Jul 15 21:26:45.431694 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:45.431447    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="903ced98-26ba-41e4-a7f7-306364f2bbba" containerName="istio-proxy"
Jul 15 21:26:45.431694 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:45.431458    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="903ced98-26ba-41e4-a7f7-306364f2bbba" containerName="istio-proxy"
Jul 15 21:26:45.431694 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:45.431532    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="903ced98-26ba-41e4-a7f7-306364f2bbba" containerName="istio-proxy"
Jul 15 21:26:45.434490 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:45.434468    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d87a1e-7b4b74bc96-p2gdn"
Jul 15 21:26:45.437983 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:45.437957    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 15 21:26:45.438140 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:45.438026    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 15 21:26:45.438140 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:45.437946    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-d87a1e-serving-cert\""
Jul 15 21:26:45.439256 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:45.439233    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-s2l9d\""
Jul 15 21:26:45.447520 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:45.447490    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d87a1e-7b4b74bc96-p2gdn"]
Jul 15 21:26:45.469041 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:45.469009    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfmv2\" (UniqueName: \"kubernetes.io/projected/1a5805c2-4824-4d49-b5c1-bb3badd7ce37-kube-api-access-zfmv2\") pod \"maas-api-e2e-inf-a-d87a1e-7b4b74bc96-p2gdn\" (UID: \"1a5805c2-4824-4d49-b5c1-bb3badd7ce37\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d87a1e-7b4b74bc96-p2gdn"
Jul 15 21:26:45.469197 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:45.469070    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a5805c2-4824-4d49-b5c1-bb3badd7ce37-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d87a1e-7b4b74bc96-p2gdn\" (UID: \"1a5805c2-4824-4d49-b5c1-bb3badd7ce37\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d87a1e-7b4b74bc96-p2gdn"
Jul 15 21:26:45.469197 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:45.469151    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a5805c2-4824-4d49-b5c1-bb3badd7ce37-maas-api-tls\") pod \"maas-api-e2e-inf-a-d87a1e-7b4b74bc96-p2gdn\" (UID: \"1a5805c2-4824-4d49-b5c1-bb3badd7ce37\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d87a1e-7b4b74bc96-p2gdn"
Jul 15 21:26:45.563169 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:45.563111    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:26:45.570240 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:45.570216    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a5805c2-4824-4d49-b5c1-bb3badd7ce37-maas-api-tls\") pod \"maas-api-e2e-inf-a-d87a1e-7b4b74bc96-p2gdn\" (UID: \"1a5805c2-4824-4d49-b5c1-bb3badd7ce37\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d87a1e-7b4b74bc96-p2gdn"
Jul 15 21:26:45.570356 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:45.570257    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfmv2\" (UniqueName: \"kubernetes.io/projected/1a5805c2-4824-4d49-b5c1-bb3badd7ce37-kube-api-access-zfmv2\") pod \"maas-api-e2e-inf-a-d87a1e-7b4b74bc96-p2gdn\" (UID: \"1a5805c2-4824-4d49-b5c1-bb3badd7ce37\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d87a1e-7b4b74bc96-p2gdn"
Jul 15 21:26:45.570356 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:45.570292    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a5805c2-4824-4d49-b5c1-bb3badd7ce37-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d87a1e-7b4b74bc96-p2gdn\" (UID: \"1a5805c2-4824-4d49-b5c1-bb3badd7ce37\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d87a1e-7b4b74bc96-p2gdn"
Jul 15 21:26:45.570488 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:26:45.570357    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-d87a1e-serving-cert: secret "maas-api-e2e-inf-a-d87a1e-serving-cert" not found
Jul 15 21:26:45.570488 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:26:45.570424    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a5805c2-4824-4d49-b5c1-bb3badd7ce37-maas-api-tls podName:1a5805c2-4824-4d49-b5c1-bb3badd7ce37 nodeName:}" failed. No retries permitted until 2026-07-15 21:26:46.070408123 +0000 UTC m=+2339.219446158 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1a5805c2-4824-4d49-b5c1-bb3badd7ce37-maas-api-tls") pod "maas-api-e2e-inf-a-d87a1e-7b4b74bc96-p2gdn" (UID: "1a5805c2-4824-4d49-b5c1-bb3badd7ce37") : secret "maas-api-e2e-inf-a-d87a1e-serving-cert" not found
Jul 15 21:26:45.570993 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:45.570976    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a5805c2-4824-4d49-b5c1-bb3badd7ce37-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d87a1e-7b4b74bc96-p2gdn\" (UID: \"1a5805c2-4824-4d49-b5c1-bb3badd7ce37\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d87a1e-7b4b74bc96-p2gdn"
Jul 15 21:26:45.583071 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:45.583044    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfmv2\" (UniqueName: \"kubernetes.io/projected/1a5805c2-4824-4d49-b5c1-bb3badd7ce37-kube-api-access-zfmv2\") pod \"maas-api-e2e-inf-a-d87a1e-7b4b74bc96-p2gdn\" (UID: \"1a5805c2-4824-4d49-b5c1-bb3badd7ce37\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d87a1e-7b4b74bc96-p2gdn"
Jul 15 21:26:45.660388 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:45.660354    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:26:45.991276 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:45.991241    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:26:46.075003 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:46.074966    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a5805c2-4824-4d49-b5c1-bb3badd7ce37-maas-api-tls\") pod \"maas-api-e2e-inf-a-d87a1e-7b4b74bc96-p2gdn\" (UID: \"1a5805c2-4824-4d49-b5c1-bb3badd7ce37\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d87a1e-7b4b74bc96-p2gdn"
Jul 15 21:26:46.077981 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:46.077953    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a5805c2-4824-4d49-b5c1-bb3badd7ce37-maas-api-tls\") pod \"maas-api-e2e-inf-a-d87a1e-7b4b74bc96-p2gdn\" (UID: \"1a5805c2-4824-4d49-b5c1-bb3badd7ce37\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d87a1e-7b4b74bc96-p2gdn"
Jul 15 21:26:46.152262 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:46.152193    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:26:46.242529 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:46.242446    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:26:46.347058 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:46.347027    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d87a1e-7b4b74bc96-p2gdn"
Jul 15 21:26:46.494480 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:46.494450    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d87a1e-7b4b74bc96-p2gdn"]
Jul 15 21:26:46.495813 ip-10-0-137-234 kubenswrapper[2566]: W0715 21:26:46.495784    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a5805c2_4824_4d49_b5c1_bb3badd7ce37.slice/crio-817f059d4d60cffb45658b4bdb6eb8ef9bf9fbb8f8f4037172fdbe1be184a155 WatchSource:0}: Error finding container 817f059d4d60cffb45658b4bdb6eb8ef9bf9fbb8f8f4037172fdbe1be184a155: Status 404 returned error can't find the container with id 817f059d4d60cffb45658b4bdb6eb8ef9bf9fbb8f8f4037172fdbe1be184a155
Jul 15 21:26:47.322849 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:47.322814    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d87a1e-7b4b74bc96-p2gdn" event={"ID":"1a5805c2-4824-4d49-b5c1-bb3badd7ce37","Type":"ContainerStarted","Data":"5b1cba23598336732a9a5af3bbe21c6d8b3a413817fa608c5a95d0d095500b3a"}
Jul 15 21:26:47.323078 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:47.322990    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d87a1e-7b4b74bc96-p2gdn"
Jul 15 21:26:47.323078 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:47.323037    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d87a1e-7b4b74bc96-p2gdn" event={"ID":"1a5805c2-4824-4d49-b5c1-bb3badd7ce37","Type":"ContainerStarted","Data":"817f059d4d60cffb45658b4bdb6eb8ef9bf9fbb8f8f4037172fdbe1be184a155"}
Jul 15 21:26:47.350803 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:47.350751    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d87a1e-7b4b74bc96-p2gdn" podStartSLOduration=1.736594002 podStartE2EDuration="2.350739023s" totalImagesPullingTime="614.145021ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:26:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:26:47.34827506 +0000 UTC m=+2340.497313116" watchObservedRunningTime="2026-07-15 21:26:47.350739023 +0000 UTC m=+2340.499777079"
Jul 15 21:26:50.083754 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:50.083718    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:26:50.307761 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:50.307727    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:26:50.636413 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:50.636374    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877"]
Jul 15 21:26:50.641714 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:50.641695    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877"
Jul 15 21:26:50.645083 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:50.645053    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-659e77-openshift-default-dockercfg-6vs7k\""
Jul 15 21:26:50.657689 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:50.657659    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877"]
Jul 15 21:26:50.674186 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:50.674163    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:26:50.724983 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:50.724948    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a27f0da-6504-4248-bd38-1f272f8c05a4-istio-data\") pod \"e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877\" (UID: \"3a27f0da-6504-4248-bd38-1f272f8c05a4\") " pod="openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877"
Jul 15 21:26:50.725162 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:50.725001    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a27f0da-6504-4248-bd38-1f272f8c05a4-workload-socket\") pod \"e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877\" (UID: \"3a27f0da-6504-4248-bd38-1f272f8c05a4\") " pod="openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877"
Jul 15 21:26:50.725162 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:50.725028    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a27f0da-6504-4248-bd38-1f272f8c05a4-workload-certs\") pod \"e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877\" (UID: \"3a27f0da-6504-4248-bd38-1f272f8c05a4\") " pod="openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877"
Jul 15 21:26:50.725162 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:50.725127    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a27f0da-6504-4248-bd38-1f272f8c05a4-credential-socket\") pod \"e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877\" (UID: \"3a27f0da-6504-4248-bd38-1f272f8c05a4\") " pod="openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877"
Jul 15 21:26:50.725284 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:50.725173    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a27f0da-6504-4248-bd38-1f272f8c05a4-istiod-ca-cert\") pod \"e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877\" (UID: \"3a27f0da-6504-4248-bd38-1f272f8c05a4\") " pod="openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877"
Jul 15 21:26:50.725284 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:50.725232    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a27f0da-6504-4248-bd38-1f272f8c05a4-istio-envoy\") pod \"e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877\" (UID: \"3a27f0da-6504-4248-bd38-1f272f8c05a4\") " pod="openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877"
Jul 15 21:26:50.725284 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:50.725263    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a27f0da-6504-4248-bd38-1f272f8c05a4-istio-token\") pod \"e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877\" (UID: \"3a27f0da-6504-4248-bd38-1f272f8c05a4\") " pod="openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877"
Jul 15 21:26:50.725389 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:50.725291    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a27f0da-6504-4248-bd38-1f272f8c05a4-istio-podinfo\") pod \"e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877\" (UID: \"3a27f0da-6504-4248-bd38-1f272f8c05a4\") " pod="openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877"
Jul 15 21:26:50.725389 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:50.725309    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfwlf\" (UniqueName: \"kubernetes.io/projected/3a27f0da-6504-4248-bd38-1f272f8c05a4-kube-api-access-mfwlf\") pod \"e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877\" (UID: \"3a27f0da-6504-4248-bd38-1f272f8c05a4\") " pod="openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877"
Jul 15 21:26:50.825998 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:50.825963    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a27f0da-6504-4248-bd38-1f272f8c05a4-workload-socket\") pod \"e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877\" (UID: \"3a27f0da-6504-4248-bd38-1f272f8c05a4\") " pod="openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877"
Jul 15 21:26:50.825998 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:50.826001    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a27f0da-6504-4248-bd38-1f272f8c05a4-workload-certs\") pod \"e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877\" (UID: \"3a27f0da-6504-4248-bd38-1f272f8c05a4\") " pod="openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877"
Jul 15 21:26:50.826253 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:50.826037    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a27f0da-6504-4248-bd38-1f272f8c05a4-credential-socket\") pod \"e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877\" (UID: \"3a27f0da-6504-4248-bd38-1f272f8c05a4\") " pod="openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877"
Jul 15 21:26:50.826253 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:50.826082    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a27f0da-6504-4248-bd38-1f272f8c05a4-istiod-ca-cert\") pod \"e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877\" (UID: \"3a27f0da-6504-4248-bd38-1f272f8c05a4\") " pod="openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877"
Jul 15 21:26:50.826253 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:50.826119    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a27f0da-6504-4248-bd38-1f272f8c05a4-istio-envoy\") pod \"e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877\" (UID: \"3a27f0da-6504-4248-bd38-1f272f8c05a4\") " pod="openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877"
Jul 15 21:26:50.826253 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:50.826158    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a27f0da-6504-4248-bd38-1f272f8c05a4-istio-token\") pod \"e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877\" (UID: \"3a27f0da-6504-4248-bd38-1f272f8c05a4\") " pod="openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877"
Jul 15 21:26:50.826253 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:50.826190    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a27f0da-6504-4248-bd38-1f272f8c05a4-istio-podinfo\") pod \"e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877\" (UID: \"3a27f0da-6504-4248-bd38-1f272f8c05a4\") " pod="openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877"
Jul 15 21:26:50.826253 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:50.826222    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfwlf\" (UniqueName: \"kubernetes.io/projected/3a27f0da-6504-4248-bd38-1f272f8c05a4-kube-api-access-mfwlf\") pod \"e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877\" (UID: \"3a27f0da-6504-4248-bd38-1f272f8c05a4\") " pod="openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877"
Jul 15 21:26:50.826556 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:50.826274    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a27f0da-6504-4248-bd38-1f272f8c05a4-istio-data\") pod \"e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877\" (UID: \"3a27f0da-6504-4248-bd38-1f272f8c05a4\") " pod="openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877"
Jul 15 21:26:50.826556 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:50.826433    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a27f0da-6504-4248-bd38-1f272f8c05a4-workload-socket\") pod \"e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877\" (UID: \"3a27f0da-6504-4248-bd38-1f272f8c05a4\") " pod="openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877"
Jul 15 21:26:50.826556 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:50.826507    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a27f0da-6504-4248-bd38-1f272f8c05a4-workload-certs\") pod \"e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877\" (UID: \"3a27f0da-6504-4248-bd38-1f272f8c05a4\") " pod="openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877"
Jul 15 21:26:50.826729 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:50.826602    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a27f0da-6504-4248-bd38-1f272f8c05a4-istio-data\") pod \"e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877\" (UID: \"3a27f0da-6504-4248-bd38-1f272f8c05a4\") " pod="openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877"
Jul 15 21:26:50.826867 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:50.826844    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a27f0da-6504-4248-bd38-1f272f8c05a4-credential-socket\") pod \"e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877\" (UID: \"3a27f0da-6504-4248-bd38-1f272f8c05a4\") " pod="openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877"
Jul 15 21:26:50.826968 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:50.826947    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a27f0da-6504-4248-bd38-1f272f8c05a4-istiod-ca-cert\") pod \"e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877\" (UID: \"3a27f0da-6504-4248-bd38-1f272f8c05a4\") " pod="openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877"
Jul 15 21:26:50.828664 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:50.828641    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a27f0da-6504-4248-bd38-1f272f8c05a4-istio-envoy\") pod \"e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877\" (UID: \"3a27f0da-6504-4248-bd38-1f272f8c05a4\") " pod="openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877"
Jul 15 21:26:50.828793 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:50.828777    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a27f0da-6504-4248-bd38-1f272f8c05a4-istio-podinfo\") pod \"e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877\" (UID: \"3a27f0da-6504-4248-bd38-1f272f8c05a4\") " pod="openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877"
Jul 15 21:26:50.835165 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:50.835140    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a27f0da-6504-4248-bd38-1f272f8c05a4-istio-token\") pod \"e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877\" (UID: \"3a27f0da-6504-4248-bd38-1f272f8c05a4\") " pod="openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877"
Jul 15 21:26:50.835314 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:50.835290    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfwlf\" (UniqueName: \"kubernetes.io/projected/3a27f0da-6504-4248-bd38-1f272f8c05a4-kube-api-access-mfwlf\") pod \"e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877\" (UID: \"3a27f0da-6504-4248-bd38-1f272f8c05a4\") " pod="openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877"
Jul 15 21:26:50.954189 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:50.954152    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877"
Jul 15 21:26:51.024606 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:51.024567    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:26:51.103697 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:51.103651    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877"]
Jul 15 21:26:51.106685 ip-10-0-137-234 kubenswrapper[2566]: W0715 21:26:51.106657    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a27f0da_6504_4248_bd38_1f272f8c05a4.slice/crio-9de9277b6afb3638643fce05d2e06d01a250e6ae9c2f150dc88552e071b4ecef WatchSource:0}: Error finding container 9de9277b6afb3638643fce05d2e06d01a250e6ae9c2f150dc88552e071b4ecef: Status 404 returned error can't find the container with id 9de9277b6afb3638643fce05d2e06d01a250e6ae9c2f150dc88552e071b4ecef
Jul 15 21:26:51.109209 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:51.109169    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:26:51.109280 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:51.109261    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:26:51.109329 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:51.109303    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:26:51.344563 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:51.344470    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877" event={"ID":"3a27f0da-6504-4248-bd38-1f272f8c05a4","Type":"ContainerStarted","Data":"3df075f0e1a9a010bf5d3a332b71ddfe10ce3c1625fae86e7096dd094c5778c4"}
Jul 15 21:26:51.344563 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:51.344538    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877" event={"ID":"3a27f0da-6504-4248-bd38-1f272f8c05a4","Type":"ContainerStarted","Data":"9de9277b6afb3638643fce05d2e06d01a250e6ae9c2f150dc88552e071b4ecef"}
Jul 15 21:26:51.369613 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:51.369549    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877" podStartSLOduration=1.369532151 podStartE2EDuration="1.369532151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:26:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:26:51.368643814 +0000 UTC m=+2344.517681895" watchObservedRunningTime="2026-07-15 21:26:51.369532151 +0000 UTC m=+2344.518570211"
Jul 15 21:26:51.954545 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:51.954493    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877"
Jul 15 21:26:52.959513 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:52.959482    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877"
Jul 15 21:26:53.333288 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:53.333211    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d87a1e-7b4b74bc96-p2gdn"
Jul 15 21:26:53.354587 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:53.354551    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877"
Jul 15 21:26:53.355444 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:53.355422    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877"
Jul 15 21:26:53.494366 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:53.494310    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:26:53.591786 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:53.591702    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-659e77-558b556c76-xlqxd"]
Jul 15 21:26:53.595498 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:53.595472    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-659e77-558b556c76-xlqxd"
Jul 15 21:26:53.598570 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:53.598546    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-659e77-serving-cert\""
Jul 15 21:26:53.603477 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:53.603199    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:26:53.611785 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:53.611099    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-659e77-558b556c76-xlqxd"]
Jul 15 21:26:53.650712 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:53.650680    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5314d10-d53f-4665-9ab5-13bbaf0e0a08-openshift-service-ca\") pod \"maas-api-e2e-inf-b-659e77-558b556c76-xlqxd\" (UID: \"d5314d10-d53f-4665-9ab5-13bbaf0e0a08\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-659e77-558b556c76-xlqxd"
Jul 15 21:26:53.650880 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:53.650722    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5314d10-d53f-4665-9ab5-13bbaf0e0a08-maas-api-tls\") pod \"maas-api-e2e-inf-b-659e77-558b556c76-xlqxd\" (UID: \"d5314d10-d53f-4665-9ab5-13bbaf0e0a08\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-659e77-558b556c76-xlqxd"
Jul 15 21:26:53.650880 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:53.650821    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctrxj\" (UniqueName: \"kubernetes.io/projected/d5314d10-d53f-4665-9ab5-13bbaf0e0a08-kube-api-access-ctrxj\") pod \"maas-api-e2e-inf-b-659e77-558b556c76-xlqxd\" (UID: \"d5314d10-d53f-4665-9ab5-13bbaf0e0a08\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-659e77-558b556c76-xlqxd"
Jul 15 21:26:53.703542 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:53.703507    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:26:53.751475 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:53.751430    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5314d10-d53f-4665-9ab5-13bbaf0e0a08-maas-api-tls\") pod \"maas-api-e2e-inf-b-659e77-558b556c76-xlqxd\" (UID: \"d5314d10-d53f-4665-9ab5-13bbaf0e0a08\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-659e77-558b556c76-xlqxd"
Jul 15 21:26:53.751665 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:53.751520    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctrxj\" (UniqueName: \"kubernetes.io/projected/d5314d10-d53f-4665-9ab5-13bbaf0e0a08-kube-api-access-ctrxj\") pod \"maas-api-e2e-inf-b-659e77-558b556c76-xlqxd\" (UID: \"d5314d10-d53f-4665-9ab5-13bbaf0e0a08\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-659e77-558b556c76-xlqxd"
Jul 15 21:26:53.751665 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:53.751593    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5314d10-d53f-4665-9ab5-13bbaf0e0a08-openshift-service-ca\") pod \"maas-api-e2e-inf-b-659e77-558b556c76-xlqxd\" (UID: \"d5314d10-d53f-4665-9ab5-13bbaf0e0a08\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-659e77-558b556c76-xlqxd"
Jul 15 21:26:53.751665 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:26:53.751595    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-659e77-serving-cert: secret "maas-api-e2e-inf-b-659e77-serving-cert" not found
Jul 15 21:26:53.751841 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:26:53.751708    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5314d10-d53f-4665-9ab5-13bbaf0e0a08-maas-api-tls podName:d5314d10-d53f-4665-9ab5-13bbaf0e0a08 nodeName:}" failed. No retries permitted until 2026-07-15 21:26:54.251684907 +0000 UTC m=+2347.400722944 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d5314d10-d53f-4665-9ab5-13bbaf0e0a08-maas-api-tls") pod "maas-api-e2e-inf-b-659e77-558b556c76-xlqxd" (UID: "d5314d10-d53f-4665-9ab5-13bbaf0e0a08") : secret "maas-api-e2e-inf-b-659e77-serving-cert" not found
Jul 15 21:26:53.752490 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:53.752462    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5314d10-d53f-4665-9ab5-13bbaf0e0a08-openshift-service-ca\") pod \"maas-api-e2e-inf-b-659e77-558b556c76-xlqxd\" (UID: \"d5314d10-d53f-4665-9ab5-13bbaf0e0a08\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-659e77-558b556c76-xlqxd"
Jul 15 21:26:53.763804 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:53.763776    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctrxj\" (UniqueName: \"kubernetes.io/projected/d5314d10-d53f-4665-9ab5-13bbaf0e0a08-kube-api-access-ctrxj\") pod \"maas-api-e2e-inf-b-659e77-558b556c76-xlqxd\" (UID: \"d5314d10-d53f-4665-9ab5-13bbaf0e0a08\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-659e77-558b556c76-xlqxd"
Jul 15 21:26:54.047122 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:54.047087    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:26:54.196638 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:54.196600    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:26:54.257936 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:54.257868    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5314d10-d53f-4665-9ab5-13bbaf0e0a08-maas-api-tls\") pod \"maas-api-e2e-inf-b-659e77-558b556c76-xlqxd\" (UID: \"d5314d10-d53f-4665-9ab5-13bbaf0e0a08\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-659e77-558b556c76-xlqxd"
Jul 15 21:26:54.260334 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:54.260316    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5314d10-d53f-4665-9ab5-13bbaf0e0a08-maas-api-tls\") pod \"maas-api-e2e-inf-b-659e77-558b556c76-xlqxd\" (UID: \"d5314d10-d53f-4665-9ab5-13bbaf0e0a08\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-659e77-558b556c76-xlqxd"
Jul 15 21:26:54.301680 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:54.301584    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:26:54.510684 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:54.510647    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-659e77-558b556c76-xlqxd"
Jul 15 21:26:54.708055 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:54.705672    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-659e77-558b556c76-xlqxd"]
Jul 15 21:26:54.710530 ip-10-0-137-234 kubenswrapper[2566]: W0715 21:26:54.710481    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5314d10_d53f_4665_9ab5_13bbaf0e0a08.slice/crio-2ec03ef6b12d4437dcb0ea717aef37a581f01a1bdfdde82da549a2ed53183d91 WatchSource:0}: Error finding container 2ec03ef6b12d4437dcb0ea717aef37a581f01a1bdfdde82da549a2ed53183d91: Status 404 returned error can't find the container with id 2ec03ef6b12d4437dcb0ea717aef37a581f01a1bdfdde82da549a2ed53183d91
Jul 15 21:26:55.366260 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:55.366220    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-659e77-558b556c76-xlqxd" event={"ID":"d5314d10-d53f-4665-9ab5-13bbaf0e0a08","Type":"ContainerStarted","Data":"c715524772e73f8db49566e585e9b18210a62130489e74ed104e682c62eb5aa0"}
Jul 15 21:26:55.366722 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:55.366274    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-659e77-558b556c76-xlqxd" event={"ID":"d5314d10-d53f-4665-9ab5-13bbaf0e0a08","Type":"ContainerStarted","Data":"2ec03ef6b12d4437dcb0ea717aef37a581f01a1bdfdde82da549a2ed53183d91"}
Jul 15 21:26:55.366722 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:55.366294    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-659e77-558b556c76-xlqxd"
Jul 15 21:26:57.922104 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:26:57.922066    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:27:00.348077 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:00.348032    2566 patch_prober.go:28] interesting pod/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 15 21:27:00.348467 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:00.348099    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r" podUID="fd207757-2b22-4a73-838e-e0ed514ee510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 15 21:27:01.376228 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:01.376131    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-659e77-558b556c76-xlqxd"
Jul 15 21:27:01.397245 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:01.397195    2566 generic.go:356] "Generic (PLEG): container finished" podID="fd207757-2b22-4a73-838e-e0ed514ee510" containerID="1ad2933093fb1225e688e1a79dfabd378b37372deb9d27aba934d92c8807e5bc" exitCode=0
Jul 15 21:27:01.397405 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:01.397252    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r" event={"ID":"fd207757-2b22-4a73-838e-e0ed514ee510","Type":"ContainerDied","Data":"1ad2933093fb1225e688e1a79dfabd378b37372deb9d27aba934d92c8807e5bc"}
Jul 15 21:27:01.397856 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:01.397830    2566 scope.go:117] "RemoveContainer" containerID="1ad2933093fb1225e688e1a79dfabd378b37372deb9d27aba934d92c8807e5bc"
Jul 15 21:27:01.399135 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:01.399106    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:27:01.399221 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:01.399200    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:27:01.399268 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:01.399255    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:27:01.399736 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:01.399697    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-659e77-558b556c76-xlqxd" podStartSLOduration=7.855729316 podStartE2EDuration="8.399684312s" totalImagesPullingTime="543.954996ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:26:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:26:55.38912147 +0000 UTC m=+2348.538159528" watchObservedRunningTime="2026-07-15 21:27:01.399684312 +0000 UTC m=+2354.548722368"
Jul 15 21:27:02.408012 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:02.407961    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r" event={"ID":"fd207757-2b22-4a73-838e-e0ed514ee510","Type":"ContainerStarted","Data":"cc123cb440d622465367385fe892173796b3ce9e62b5c27845e4ca8dd70c692b"}
Jul 15 21:27:03.312980 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:03.312938    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r"
Jul 15 21:27:03.313221 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:03.313082    2566 patch_prober.go:28] interesting pod/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Jul 15 21:27:03.313221 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:03.313146    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r" podUID="fd207757-2b22-4a73-838e-e0ed514ee510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Jul 15 21:27:04.313044 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:04.312998    2566 patch_prober.go:28] interesting pod/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Jul 15 21:27:04.313569 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:04.313098    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r" podUID="fd207757-2b22-4a73-838e-e0ed514ee510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Jul 15 21:27:05.313379 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:05.313339    2566 patch_prober.go:28] interesting pod/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Jul 15 21:27:05.314016 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:05.313970    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r" podUID="fd207757-2b22-4a73-838e-e0ed514ee510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Jul 15 21:27:06.313781 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:06.313736    2566 patch_prober.go:28] interesting pod/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Jul 15 21:27:06.314229 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:06.313802    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r" podUID="fd207757-2b22-4a73-838e-e0ed514ee510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Jul 15 21:27:07.133384 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:07.133346    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:27:07.231037 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:07.230982    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:27:07.337059 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:07.337030    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r"
Jul 15 21:27:07.337428 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:07.337405    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r"
Jul 15 21:27:07.338141 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:07.338120    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r"
Jul 15 21:27:09.397049 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:09.397004    2566 patch_prober.go:28] interesting pod/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 15 21:27:09.397499 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:09.397069    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877" podUID="3a27f0da-6504-4248-bd38-1f272f8c05a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 15 21:27:10.457584 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:10.457548    2566 generic.go:356] "Generic (PLEG): container finished" podID="3a27f0da-6504-4248-bd38-1f272f8c05a4" containerID="3df075f0e1a9a010bf5d3a332b71ddfe10ce3c1625fae86e7096dd094c5778c4" exitCode=0
Jul 15 21:27:10.458144 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:10.457610    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877" event={"ID":"3a27f0da-6504-4248-bd38-1f272f8c05a4","Type":"ContainerDied","Data":"3df075f0e1a9a010bf5d3a332b71ddfe10ce3c1625fae86e7096dd094c5778c4"}
Jul 15 21:27:10.458219 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:10.458171    2566 scope.go:117] "RemoveContainer" containerID="3df075f0e1a9a010bf5d3a332b71ddfe10ce3c1625fae86e7096dd094c5778c4"
Jul 15 21:27:10.459216 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:10.459186    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:27:10.459319 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:10.459255    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:27:10.459319 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:10.459286    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:27:11.464241 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:11.464200    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877" event={"ID":"3a27f0da-6504-4248-bd38-1f272f8c05a4","Type":"ContainerStarted","Data":"185b33f6c9a6c7cac5a2a53a3f16d33118aa81e7f9f038e931bde11d259604b1"}
Jul 15 21:27:11.954966 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:11.954932    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877"
Jul 15 21:27:11.955208 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:11.955180    2566 patch_prober.go:28] interesting pod/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Jul 15 21:27:11.955254 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:11.955238    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877" podUID="3a27f0da-6504-4248-bd38-1f272f8c05a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Jul 15 21:27:12.955002 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:12.954966    2566 patch_prober.go:28] interesting pod/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Jul 15 21:27:12.955390 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:12.955025    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877" podUID="3a27f0da-6504-4248-bd38-1f272f8c05a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Jul 15 21:27:13.954710 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:13.954673    2566 patch_prober.go:28] interesting pod/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Jul 15 21:27:13.954887 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:13.954728    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877" podUID="3a27f0da-6504-4248-bd38-1f272f8c05a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Jul 15 21:27:14.954853 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:14.954817    2566 patch_prober.go:28] interesting pod/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Jul 15 21:27:14.955319 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:14.954874    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877" podUID="3a27f0da-6504-4248-bd38-1f272f8c05a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Jul 15 21:27:15.121088 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:15.121053    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:27:15.221016 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:15.220932    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:27:15.958296 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:15.958267    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877"
Jul 15 21:27:15.958769 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:15.958570    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877"
Jul 15 21:27:15.959369 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:15.959352    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877"
Jul 15 21:27:21.785806 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:21.785774    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:27:32.859787 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:32.859737    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:27:40.341650 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:40.341610    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:27:47.829500 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:47.829470    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-652lk_f9a159bb-39c4-4cca-92ff-1c74d739e311/ovn-acl-logging/0.log"
Jul 15 21:27:47.831509 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:47.831485    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-652lk_f9a159bb-39c4-4cca-92ff-1c74d739e311/ovn-acl-logging/0.log"
Jul 15 21:27:54.238951 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:54.238915    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:27:57.320201 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:57.320167    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:27:57.425493 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:27:57.425449    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:28:12.745015 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:12.744928    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:28:13.776922 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:13.776870    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:28:14.536596 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:14.536536    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:28:14.633435 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:14.633400    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:28:18.516837 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:18.516795    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:28:24.420988 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:24.420954    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d87a1e-7b4b74bc96-p2gdn"]
Jul 15 21:28:24.421478 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:24.421175    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d87a1e-7b4b74bc96-p2gdn" podUID="1a5805c2-4824-4d49-b5c1-bb3badd7ce37" containerName="maas-api" containerID="cri-o://5b1cba23598336732a9a5af3bbe21c6d8b3a413817fa608c5a95d0d095500b3a" gracePeriod=30
Jul 15 21:28:24.459620 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:24.459592    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:28:24.669012 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:24.668987    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d87a1e-7b4b74bc96-p2gdn"
Jul 15 21:28:24.698444 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:24.698414    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a5805c2-4824-4d49-b5c1-bb3badd7ce37-openshift-service-ca\") pod \"1a5805c2-4824-4d49-b5c1-bb3badd7ce37\" (UID: \"1a5805c2-4824-4d49-b5c1-bb3badd7ce37\") "
Jul 15 21:28:24.698795 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:24.698453    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfmv2\" (UniqueName: \"kubernetes.io/projected/1a5805c2-4824-4d49-b5c1-bb3badd7ce37-kube-api-access-zfmv2\") pod \"1a5805c2-4824-4d49-b5c1-bb3badd7ce37\" (UID: \"1a5805c2-4824-4d49-b5c1-bb3badd7ce37\") "
Jul 15 21:28:24.698795 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:24.698498    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a5805c2-4824-4d49-b5c1-bb3badd7ce37-maas-api-tls\") pod \"1a5805c2-4824-4d49-b5c1-bb3badd7ce37\" (UID: \"1a5805c2-4824-4d49-b5c1-bb3badd7ce37\") "
Jul 15 21:28:24.698795 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:24.698765    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a5805c2-4824-4d49-b5c1-bb3badd7ce37-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1a5805c2-4824-4d49-b5c1-bb3badd7ce37" (UID: "1a5805c2-4824-4d49-b5c1-bb3badd7ce37"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 21:28:24.700709 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:24.700671    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a5805c2-4824-4d49-b5c1-bb3badd7ce37-kube-api-access-zfmv2" (OuterVolumeSpecName: "kube-api-access-zfmv2") pod "1a5805c2-4824-4d49-b5c1-bb3badd7ce37" (UID: "1a5805c2-4824-4d49-b5c1-bb3badd7ce37"). InnerVolumeSpecName "kube-api-access-zfmv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 21:28:24.700823 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:24.700723    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a5805c2-4824-4d49-b5c1-bb3badd7ce37-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1a5805c2-4824-4d49-b5c1-bb3badd7ce37" (UID: "1a5805c2-4824-4d49-b5c1-bb3badd7ce37"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 21:28:24.799959 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:24.799923    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a5805c2-4824-4d49-b5c1-bb3badd7ce37-maas-api-tls\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:28:24.799959 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:24.799954    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a5805c2-4824-4d49-b5c1-bb3badd7ce37-openshift-service-ca\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:28:24.799959 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:24.799963    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfmv2\" (UniqueName: \"kubernetes.io/projected/1a5805c2-4824-4d49-b5c1-bb3badd7ce37-kube-api-access-zfmv2\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:28:24.826553 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:24.826519    2566 generic.go:356] "Generic (PLEG): container finished" podID="1a5805c2-4824-4d49-b5c1-bb3badd7ce37" containerID="5b1cba23598336732a9a5af3bbe21c6d8b3a413817fa608c5a95d0d095500b3a" exitCode=0
Jul 15 21:28:24.826737 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:24.826563    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d87a1e-7b4b74bc96-p2gdn"
Jul 15 21:28:24.826737 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:24.826611    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d87a1e-7b4b74bc96-p2gdn" event={"ID":"1a5805c2-4824-4d49-b5c1-bb3badd7ce37","Type":"ContainerDied","Data":"5b1cba23598336732a9a5af3bbe21c6d8b3a413817fa608c5a95d0d095500b3a"}
Jul 15 21:28:24.826737 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:24.826667    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d87a1e-7b4b74bc96-p2gdn" event={"ID":"1a5805c2-4824-4d49-b5c1-bb3badd7ce37","Type":"ContainerDied","Data":"817f059d4d60cffb45658b4bdb6eb8ef9bf9fbb8f8f4037172fdbe1be184a155"}
Jul 15 21:28:24.826737 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:24.826696    2566 scope.go:117] "RemoveContainer" containerID="5b1cba23598336732a9a5af3bbe21c6d8b3a413817fa608c5a95d0d095500b3a"
Jul 15 21:28:24.836019 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:24.836001    2566 scope.go:117] "RemoveContainer" containerID="5b1cba23598336732a9a5af3bbe21c6d8b3a413817fa608c5a95d0d095500b3a"
Jul 15 21:28:24.836288 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:28:24.836268    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b1cba23598336732a9a5af3bbe21c6d8b3a413817fa608c5a95d0d095500b3a\": container with ID starting with 5b1cba23598336732a9a5af3bbe21c6d8b3a413817fa608c5a95d0d095500b3a not found: ID does not exist" containerID="5b1cba23598336732a9a5af3bbe21c6d8b3a413817fa608c5a95d0d095500b3a"
Jul 15 21:28:24.836341 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:24.836297    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b1cba23598336732a9a5af3bbe21c6d8b3a413817fa608c5a95d0d095500b3a"} err="failed to get container status \"5b1cba23598336732a9a5af3bbe21c6d8b3a413817fa608c5a95d0d095500b3a\": rpc error: code = NotFound desc = could not find container \"5b1cba23598336732a9a5af3bbe21c6d8b3a413817fa608c5a95d0d095500b3a\": container with ID starting with 5b1cba23598336732a9a5af3bbe21c6d8b3a413817fa608c5a95d0d095500b3a not found: ID does not exist"
Jul 15 21:28:24.850040 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:24.850009    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d87a1e-7b4b74bc96-p2gdn"]
Jul 15 21:28:24.853725 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:24.853704    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d87a1e-7b4b74bc96-p2gdn"]
Jul 15 21:28:25.756453 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:25.756414    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a5805c2-4824-4d49-b5c1-bb3badd7ce37" path="/var/lib/kubelet/pods/1a5805c2-4824-4d49-b5c1-bb3badd7ce37/volumes"
Jul 15 21:28:28.994847 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:28.994806    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:28:35.669433 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:35.669396    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:28:35.753229 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:35.753195    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:28:39.761177 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:39.761144    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r"]
Jul 15 21:28:39.761617 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:39.761346    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r" podUID="fd207757-2b22-4a73-838e-e0ed514ee510" containerName="istio-proxy" containerID="cri-o://cc123cb440d622465367385fe892173796b3ce9e62b5c27845e4ca8dd70c692b" gracePeriod=30
Jul 15 21:28:40.243048 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:40.242979    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:28:40.342915 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:40.342857    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:28:40.997183 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:40.997149    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:28:41.097327 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:41.097290    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:28:44.934051 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:44.934023    2566 generic.go:356] "Generic (PLEG): container finished" podID="fd207757-2b22-4a73-838e-e0ed514ee510" containerID="cc123cb440d622465367385fe892173796b3ce9e62b5c27845e4ca8dd70c692b" exitCode=0
Jul 15 21:28:44.934410 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:44.934066    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r" event={"ID":"fd207757-2b22-4a73-838e-e0ed514ee510","Type":"ContainerDied","Data":"cc123cb440d622465367385fe892173796b3ce9e62b5c27845e4ca8dd70c692b"}
Jul 15 21:28:44.934410 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:44.934121    2566 scope.go:117] "RemoveContainer" containerID="1ad2933093fb1225e688e1a79dfabd378b37372deb9d27aba934d92c8807e5bc"
Jul 15 21:28:45.017905 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:45.017872    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r"
Jul 15 21:28:45.187778 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:45.187690    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd207757-2b22-4a73-838e-e0ed514ee510-istio-token\") pod \"fd207757-2b22-4a73-838e-e0ed514ee510\" (UID: \"fd207757-2b22-4a73-838e-e0ed514ee510\") "
Jul 15 21:28:45.187778 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:45.187738    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd207757-2b22-4a73-838e-e0ed514ee510-credential-socket\") pod \"fd207757-2b22-4a73-838e-e0ed514ee510\" (UID: \"fd207757-2b22-4a73-838e-e0ed514ee510\") "
Jul 15 21:28:45.187778 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:45.187769    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd207757-2b22-4a73-838e-e0ed514ee510-workload-socket\") pod \"fd207757-2b22-4a73-838e-e0ed514ee510\" (UID: \"fd207757-2b22-4a73-838e-e0ed514ee510\") "
Jul 15 21:28:45.188044 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:45.187882    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2ppn\" (UniqueName: \"kubernetes.io/projected/fd207757-2b22-4a73-838e-e0ed514ee510-kube-api-access-v2ppn\") pod \"fd207757-2b22-4a73-838e-e0ed514ee510\" (UID: \"fd207757-2b22-4a73-838e-e0ed514ee510\") "
Jul 15 21:28:45.188044 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:45.187942    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd207757-2b22-4a73-838e-e0ed514ee510-istio-podinfo\") pod \"fd207757-2b22-4a73-838e-e0ed514ee510\" (UID: \"fd207757-2b22-4a73-838e-e0ed514ee510\") "
Jul 15 21:28:45.188044 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:45.187979    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd207757-2b22-4a73-838e-e0ed514ee510-istio-envoy\") pod \"fd207757-2b22-4a73-838e-e0ed514ee510\" (UID: \"fd207757-2b22-4a73-838e-e0ed514ee510\") "
Jul 15 21:28:45.188206 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:45.188061    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd207757-2b22-4a73-838e-e0ed514ee510-istio-data\") pod \"fd207757-2b22-4a73-838e-e0ed514ee510\" (UID: \"fd207757-2b22-4a73-838e-e0ed514ee510\") "
Jul 15 21:28:45.188206 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:45.188068    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd207757-2b22-4a73-838e-e0ed514ee510-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fd207757-2b22-4a73-838e-e0ed514ee510" (UID: "fd207757-2b22-4a73-838e-e0ed514ee510"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:28:45.188206 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:45.188088    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd207757-2b22-4a73-838e-e0ed514ee510-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fd207757-2b22-4a73-838e-e0ed514ee510" (UID: "fd207757-2b22-4a73-838e-e0ed514ee510"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:28:45.188206 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:45.188105    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd207757-2b22-4a73-838e-e0ed514ee510-istiod-ca-cert\") pod \"fd207757-2b22-4a73-838e-e0ed514ee510\" (UID: \"fd207757-2b22-4a73-838e-e0ed514ee510\") "
Jul 15 21:28:45.188206 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:45.188134    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd207757-2b22-4a73-838e-e0ed514ee510-workload-certs\") pod \"fd207757-2b22-4a73-838e-e0ed514ee510\" (UID: \"fd207757-2b22-4a73-838e-e0ed514ee510\") "
Jul 15 21:28:45.188476 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:45.188408    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd207757-2b22-4a73-838e-e0ed514ee510-credential-socket\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:28:45.188476 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:45.188430    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd207757-2b22-4a73-838e-e0ed514ee510-workload-socket\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:28:45.188476 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:45.188442    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd207757-2b22-4a73-838e-e0ed514ee510-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fd207757-2b22-4a73-838e-e0ed514ee510" (UID: "fd207757-2b22-4a73-838e-e0ed514ee510"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 21:28:45.188640 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:45.188613    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd207757-2b22-4a73-838e-e0ed514ee510-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fd207757-2b22-4a73-838e-e0ed514ee510" (UID: "fd207757-2b22-4a73-838e-e0ed514ee510"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:28:45.188952 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:45.188932    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd207757-2b22-4a73-838e-e0ed514ee510-istio-data" (OuterVolumeSpecName: "istio-data") pod "fd207757-2b22-4a73-838e-e0ed514ee510" (UID: "fd207757-2b22-4a73-838e-e0ed514ee510"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:28:45.190067 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:45.190043    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd207757-2b22-4a73-838e-e0ed514ee510-istio-token" (OuterVolumeSpecName: "istio-token") pod "fd207757-2b22-4a73-838e-e0ed514ee510" (UID: "fd207757-2b22-4a73-838e-e0ed514ee510"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 21:28:45.190215 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:45.190199    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd207757-2b22-4a73-838e-e0ed514ee510-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fd207757-2b22-4a73-838e-e0ed514ee510" (UID: "fd207757-2b22-4a73-838e-e0ed514ee510"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 15 21:28:45.190427 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:45.190410    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd207757-2b22-4a73-838e-e0ed514ee510-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fd207757-2b22-4a73-838e-e0ed514ee510" (UID: "fd207757-2b22-4a73-838e-e0ed514ee510"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:28:45.190471 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:45.190447    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd207757-2b22-4a73-838e-e0ed514ee510-kube-api-access-v2ppn" (OuterVolumeSpecName: "kube-api-access-v2ppn") pod "fd207757-2b22-4a73-838e-e0ed514ee510" (UID: "fd207757-2b22-4a73-838e-e0ed514ee510"). InnerVolumeSpecName "kube-api-access-v2ppn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 21:28:45.289165 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:45.289137    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2ppn\" (UniqueName: \"kubernetes.io/projected/fd207757-2b22-4a73-838e-e0ed514ee510-kube-api-access-v2ppn\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:28:45.289165 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:45.289164    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd207757-2b22-4a73-838e-e0ed514ee510-istio-podinfo\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:28:45.289373 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:45.289174    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd207757-2b22-4a73-838e-e0ed514ee510-istio-envoy\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:28:45.289373 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:45.289183    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd207757-2b22-4a73-838e-e0ed514ee510-istio-data\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:28:45.289373 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:45.289190    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd207757-2b22-4a73-838e-e0ed514ee510-istiod-ca-cert\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:28:45.289373 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:45.289198    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd207757-2b22-4a73-838e-e0ed514ee510-workload-certs\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:28:45.289373 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:45.289206    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd207757-2b22-4a73-838e-e0ed514ee510-istio-token\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:28:45.939699 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:45.939673    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r"
Jul 15 21:28:45.940211 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:45.939672    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r" event={"ID":"fd207757-2b22-4a73-838e-e0ed514ee510","Type":"ContainerDied","Data":"a27e1455f133e7cdc4b58c15a97ea5606e2bfb46024ab7f5eb6a155be0b15365"}
Jul 15 21:28:45.940211 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:45.939808    2566 scope.go:117] "RemoveContainer" containerID="cc123cb440d622465367385fe892173796b3ce9e62b5c27845e4ca8dd70c692b"
Jul 15 21:28:45.961279 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:45.961247    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r"]
Jul 15 21:28:45.967997 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:45.964877    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d87a1e-openshift-default-5dd6c4b5b9-9j69r"]
Jul 15 21:28:47.756950 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:47.756889    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd207757-2b22-4a73-838e-e0ed514ee510" path="/var/lib/kubelet/pods/fd207757-2b22-4a73-838e-e0ed514ee510/volumes"
Jul 15 21:28:48.566178 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:48.566142    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:28:50.669796 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:50.669736    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-659e77-558b556c76-xlqxd"]
Jul 15 21:28:50.670176 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:50.670022    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-659e77-558b556c76-xlqxd" podUID="d5314d10-d53f-4665-9ab5-13bbaf0e0a08" containerName="maas-api" containerID="cri-o://c715524772e73f8db49566e585e9b18210a62130489e74ed104e682c62eb5aa0" gracePeriod=30
Jul 15 21:28:50.701531 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:50.701500    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:28:50.921878 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:50.921813    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-659e77-558b556c76-xlqxd"
Jul 15 21:28:50.965405 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:50.965374    2566 generic.go:356] "Generic (PLEG): container finished" podID="d5314d10-d53f-4665-9ab5-13bbaf0e0a08" containerID="c715524772e73f8db49566e585e9b18210a62130489e74ed104e682c62eb5aa0" exitCode=0
Jul 15 21:28:50.965568 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:50.965421    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-659e77-558b556c76-xlqxd"
Jul 15 21:28:50.965568 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:50.965460    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-659e77-558b556c76-xlqxd" event={"ID":"d5314d10-d53f-4665-9ab5-13bbaf0e0a08","Type":"ContainerDied","Data":"c715524772e73f8db49566e585e9b18210a62130489e74ed104e682c62eb5aa0"}
Jul 15 21:28:50.965568 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:50.965514    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-659e77-558b556c76-xlqxd" event={"ID":"d5314d10-d53f-4665-9ab5-13bbaf0e0a08","Type":"ContainerDied","Data":"2ec03ef6b12d4437dcb0ea717aef37a581f01a1bdfdde82da549a2ed53183d91"}
Jul 15 21:28:50.965568 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:50.965540    2566 scope.go:117] "RemoveContainer" containerID="c715524772e73f8db49566e585e9b18210a62130489e74ed104e682c62eb5aa0"
Jul 15 21:28:50.974022 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:50.974003    2566 scope.go:117] "RemoveContainer" containerID="c715524772e73f8db49566e585e9b18210a62130489e74ed104e682c62eb5aa0"
Jul 15 21:28:50.974285 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:28:50.974266    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c715524772e73f8db49566e585e9b18210a62130489e74ed104e682c62eb5aa0\": container with ID starting with c715524772e73f8db49566e585e9b18210a62130489e74ed104e682c62eb5aa0 not found: ID does not exist" containerID="c715524772e73f8db49566e585e9b18210a62130489e74ed104e682c62eb5aa0"
Jul 15 21:28:50.974376 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:50.974293    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c715524772e73f8db49566e585e9b18210a62130489e74ed104e682c62eb5aa0"} err="failed to get container status \"c715524772e73f8db49566e585e9b18210a62130489e74ed104e682c62eb5aa0\": rpc error: code = NotFound desc = could not find container \"c715524772e73f8db49566e585e9b18210a62130489e74ed104e682c62eb5aa0\": container with ID starting with c715524772e73f8db49566e585e9b18210a62130489e74ed104e682c62eb5aa0 not found: ID does not exist"
Jul 15 21:28:51.038071 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:51.038035    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5314d10-d53f-4665-9ab5-13bbaf0e0a08-maas-api-tls\") pod \"d5314d10-d53f-4665-9ab5-13bbaf0e0a08\" (UID: \"d5314d10-d53f-4665-9ab5-13bbaf0e0a08\") "
Jul 15 21:28:51.038071 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:51.038072    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctrxj\" (UniqueName: \"kubernetes.io/projected/d5314d10-d53f-4665-9ab5-13bbaf0e0a08-kube-api-access-ctrxj\") pod \"d5314d10-d53f-4665-9ab5-13bbaf0e0a08\" (UID: \"d5314d10-d53f-4665-9ab5-13bbaf0e0a08\") "
Jul 15 21:28:51.038302 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:51.038125    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5314d10-d53f-4665-9ab5-13bbaf0e0a08-openshift-service-ca\") pod \"d5314d10-d53f-4665-9ab5-13bbaf0e0a08\" (UID: \"d5314d10-d53f-4665-9ab5-13bbaf0e0a08\") "
Jul 15 21:28:51.038577 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:51.038544    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5314d10-d53f-4665-9ab5-13bbaf0e0a08-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d5314d10-d53f-4665-9ab5-13bbaf0e0a08" (UID: "d5314d10-d53f-4665-9ab5-13bbaf0e0a08"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 21:28:51.040121 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:51.040101    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5314d10-d53f-4665-9ab5-13bbaf0e0a08-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d5314d10-d53f-4665-9ab5-13bbaf0e0a08" (UID: "d5314d10-d53f-4665-9ab5-13bbaf0e0a08"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 21:28:51.040299 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:51.040279    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5314d10-d53f-4665-9ab5-13bbaf0e0a08-kube-api-access-ctrxj" (OuterVolumeSpecName: "kube-api-access-ctrxj") pod "d5314d10-d53f-4665-9ab5-13bbaf0e0a08" (UID: "d5314d10-d53f-4665-9ab5-13bbaf0e0a08"). InnerVolumeSpecName "kube-api-access-ctrxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 21:28:51.139312 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:51.139269    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5314d10-d53f-4665-9ab5-13bbaf0e0a08-maas-api-tls\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:28:51.139312 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:51.139306    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctrxj\" (UniqueName: \"kubernetes.io/projected/d5314d10-d53f-4665-9ab5-13bbaf0e0a08-kube-api-access-ctrxj\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:28:51.139520 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:51.139322    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5314d10-d53f-4665-9ab5-13bbaf0e0a08-openshift-service-ca\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:28:51.295927 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:51.295484    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-659e77-558b556c76-xlqxd"]
Jul 15 21:28:51.300534 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:51.300493    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-659e77-558b556c76-xlqxd"]
Jul 15 21:28:51.755961 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:51.755926    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5314d10-d53f-4665-9ab5-13bbaf0e0a08" path="/var/lib/kubelet/pods/d5314d10-d53f-4665-9ab5-13bbaf0e0a08/volumes"
Jul 15 21:28:55.250178 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:28:55.250147    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:29:01.931129 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:01.931091    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:29:02.258223 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:02.258187    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:29:06.025958 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:06.025927    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877"]
Jul 15 21:29:06.026337 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:06.026135    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877" podUID="3a27f0da-6504-4248-bd38-1f272f8c05a4" containerName="istio-proxy" containerID="cri-o://185b33f6c9a6c7cac5a2a53a3f16d33118aa81e7f9f038e931bde11d259604b1" gracePeriod=30
Jul 15 21:29:06.035400 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:06.035363    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:29:06.268991 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:06.268951    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:29:06.644310 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:06.644273    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:29:07.098658 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:07.098616    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k"]
Jul 15 21:29:07.099062 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:07.099012    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd207757-2b22-4a73-838e-e0ed514ee510" containerName="istio-proxy"
Jul 15 21:29:07.099062 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:07.099024    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd207757-2b22-4a73-838e-e0ed514ee510" containerName="istio-proxy"
Jul 15 21:29:07.099062 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:07.099037    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a5805c2-4824-4d49-b5c1-bb3badd7ce37" containerName="maas-api"
Jul 15 21:29:07.099062 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:07.099046    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a5805c2-4824-4d49-b5c1-bb3badd7ce37" containerName="maas-api"
Jul 15 21:29:07.099062 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:07.099053    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5314d10-d53f-4665-9ab5-13bbaf0e0a08" containerName="maas-api"
Jul 15 21:29:07.099062 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:07.099058    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5314d10-d53f-4665-9ab5-13bbaf0e0a08" containerName="maas-api"
Jul 15 21:29:07.099259 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:07.099116    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd207757-2b22-4a73-838e-e0ed514ee510" containerName="istio-proxy"
Jul 15 21:29:07.099259 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:07.099127    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd207757-2b22-4a73-838e-e0ed514ee510" containerName="istio-proxy"
Jul 15 21:29:07.099259 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:07.099135    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a5805c2-4824-4d49-b5c1-bb3badd7ce37" containerName="maas-api"
Jul 15 21:29:07.099259 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:07.099141    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5314d10-d53f-4665-9ab5-13bbaf0e0a08" containerName="maas-api"
Jul 15 21:29:07.101105 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:07.101086    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k"
Jul 15 21:29:07.104815 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:07.104797    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f5fcf1-openshift-default-dockercfg-sd7w8\""
Jul 15 21:29:07.128284 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:07.128254    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k"]
Jul 15 21:29:07.184989 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:07.184960    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/312dafc1-a9a9-44ae-9288-de82c7fb439f-credential-socket\") pod \"e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k\" (UID: \"312dafc1-a9a9-44ae-9288-de82c7fb439f\") " pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k"
Jul 15 21:29:07.185182 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:07.185022    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/312dafc1-a9a9-44ae-9288-de82c7fb439f-istio-envoy\") pod \"e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k\" (UID: \"312dafc1-a9a9-44ae-9288-de82c7fb439f\") " pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k"
Jul 15 21:29:07.185182 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:07.185055    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/312dafc1-a9a9-44ae-9288-de82c7fb439f-istio-token\") pod \"e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k\" (UID: \"312dafc1-a9a9-44ae-9288-de82c7fb439f\") " pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k"
Jul 15 21:29:07.185182 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:07.185086    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/312dafc1-a9a9-44ae-9288-de82c7fb439f-istio-podinfo\") pod \"e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k\" (UID: \"312dafc1-a9a9-44ae-9288-de82c7fb439f\") " pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k"
Jul 15 21:29:07.185182 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:07.185112    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/312dafc1-a9a9-44ae-9288-de82c7fb439f-istiod-ca-cert\") pod \"e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k\" (UID: \"312dafc1-a9a9-44ae-9288-de82c7fb439f\") " pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k"
Jul 15 21:29:07.185182 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:07.185144    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/312dafc1-a9a9-44ae-9288-de82c7fb439f-workload-socket\") pod \"e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k\" (UID: \"312dafc1-a9a9-44ae-9288-de82c7fb439f\") " pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k"
Jul 15 21:29:07.185182 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:07.185166    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/312dafc1-a9a9-44ae-9288-de82c7fb439f-istio-data\") pod \"e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k\" (UID: \"312dafc1-a9a9-44ae-9288-de82c7fb439f\") " pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k"
Jul 15 21:29:07.185561 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:07.185184    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hdlm\" (UniqueName: \"kubernetes.io/projected/312dafc1-a9a9-44ae-9288-de82c7fb439f-kube-api-access-2hdlm\") pod \"e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k\" (UID: \"312dafc1-a9a9-44ae-9288-de82c7fb439f\") " pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k"
Jul 15 21:29:07.185561 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:07.185241    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/312dafc1-a9a9-44ae-9288-de82c7fb439f-workload-certs\") pod \"e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k\" (UID: \"312dafc1-a9a9-44ae-9288-de82c7fb439f\") " pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k"
Jul 15 21:29:07.286032 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:07.285995    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/312dafc1-a9a9-44ae-9288-de82c7fb439f-credential-socket\") pod \"e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k\" (UID: \"312dafc1-a9a9-44ae-9288-de82c7fb439f\") " pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k"
Jul 15 21:29:07.286247 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:07.286059    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/312dafc1-a9a9-44ae-9288-de82c7fb439f-istio-envoy\") pod \"e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k\" (UID: \"312dafc1-a9a9-44ae-9288-de82c7fb439f\") " pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k"
Jul 15 21:29:07.286247 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:07.286082    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/312dafc1-a9a9-44ae-9288-de82c7fb439f-istio-token\") pod \"e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k\" (UID: \"312dafc1-a9a9-44ae-9288-de82c7fb439f\") " pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k"
Jul 15 21:29:07.286247 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:07.286112    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/312dafc1-a9a9-44ae-9288-de82c7fb439f-istio-podinfo\") pod \"e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k\" (UID: \"312dafc1-a9a9-44ae-9288-de82c7fb439f\") " pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k"
Jul 15 21:29:07.286247 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:07.286131    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/312dafc1-a9a9-44ae-9288-de82c7fb439f-istiod-ca-cert\") pod \"e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k\" (UID: \"312dafc1-a9a9-44ae-9288-de82c7fb439f\") " pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k"
Jul 15 21:29:07.286247 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:07.286161    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/312dafc1-a9a9-44ae-9288-de82c7fb439f-workload-socket\") pod \"e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k\" (UID: \"312dafc1-a9a9-44ae-9288-de82c7fb439f\") " pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k"
Jul 15 21:29:07.286247 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:07.286185    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/312dafc1-a9a9-44ae-9288-de82c7fb439f-istio-data\") pod \"e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k\" (UID: \"312dafc1-a9a9-44ae-9288-de82c7fb439f\") " pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k"
Jul 15 21:29:07.286247 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:07.286214    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hdlm\" (UniqueName: \"kubernetes.io/projected/312dafc1-a9a9-44ae-9288-de82c7fb439f-kube-api-access-2hdlm\") pod \"e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k\" (UID: \"312dafc1-a9a9-44ae-9288-de82c7fb439f\") " pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k"
Jul 15 21:29:07.286645 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:07.286252    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/312dafc1-a9a9-44ae-9288-de82c7fb439f-workload-certs\") pod \"e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k\" (UID: \"312dafc1-a9a9-44ae-9288-de82c7fb439f\") " pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k"
Jul 15 21:29:07.286645 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:07.286488    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/312dafc1-a9a9-44ae-9288-de82c7fb439f-credential-socket\") pod \"e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k\" (UID: \"312dafc1-a9a9-44ae-9288-de82c7fb439f\") " pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k"
Jul 15 21:29:07.286645 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:07.286564    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/312dafc1-a9a9-44ae-9288-de82c7fb439f-workload-certs\") pod \"e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k\" (UID: \"312dafc1-a9a9-44ae-9288-de82c7fb439f\") " pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k"
Jul 15 21:29:07.286798 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:07.286666    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/312dafc1-a9a9-44ae-9288-de82c7fb439f-workload-socket\") pod \"e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k\" (UID: \"312dafc1-a9a9-44ae-9288-de82c7fb439f\") " pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k"
Jul 15 21:29:07.286873 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:07.286846    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/312dafc1-a9a9-44ae-9288-de82c7fb439f-istio-data\") pod \"e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k\" (UID: \"312dafc1-a9a9-44ae-9288-de82c7fb439f\") " pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k"
Jul 15 21:29:07.287060 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:07.287040    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/312dafc1-a9a9-44ae-9288-de82c7fb439f-istiod-ca-cert\") pod \"e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k\" (UID: \"312dafc1-a9a9-44ae-9288-de82c7fb439f\") " pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k"
Jul 15 21:29:07.288457 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:07.288434    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/312dafc1-a9a9-44ae-9288-de82c7fb439f-istio-envoy\") pod \"e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k\" (UID: \"312dafc1-a9a9-44ae-9288-de82c7fb439f\") " pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k"
Jul 15 21:29:07.288780 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:07.288751    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/312dafc1-a9a9-44ae-9288-de82c7fb439f-istio-podinfo\") pod \"e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k\" (UID: \"312dafc1-a9a9-44ae-9288-de82c7fb439f\") " pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k"
Jul 15 21:29:07.301714 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:07.301682    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/312dafc1-a9a9-44ae-9288-de82c7fb439f-istio-token\") pod \"e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k\" (UID: \"312dafc1-a9a9-44ae-9288-de82c7fb439f\") " pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k"
Jul 15 21:29:07.301865 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:07.301762    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hdlm\" (UniqueName: \"kubernetes.io/projected/312dafc1-a9a9-44ae-9288-de82c7fb439f-kube-api-access-2hdlm\") pod \"e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k\" (UID: \"312dafc1-a9a9-44ae-9288-de82c7fb439f\") " pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k"
Jul 15 21:29:07.412010 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:07.411912    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k"
Jul 15 21:29:07.452218 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:07.451999    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:29:07.563013 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:07.562975    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k"]
Jul 15 21:29:07.566112 ip-10-0-137-234 kubenswrapper[2566]: W0715 21:29:07.566085    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod312dafc1_a9a9_44ae_9288_de82c7fb439f.slice/crio-1ba84374f6c990ded86acdd9461a3ce1c36f47d07a5f542363651c777b4434ee WatchSource:0}: Error finding container 1ba84374f6c990ded86acdd9461a3ce1c36f47d07a5f542363651c777b4434ee: Status 404 returned error can't find the container with id 1ba84374f6c990ded86acdd9461a3ce1c36f47d07a5f542363651c777b4434ee
Jul 15 21:29:07.568354 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:07.568319    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:29:07.568462 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:07.568396    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:29:07.568462 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:07.568424    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:29:07.768889 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:07.768854    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:29:08.050625 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:08.050521    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k" event={"ID":"312dafc1-a9a9-44ae-9288-de82c7fb439f","Type":"ContainerStarted","Data":"514b3d940dd98131d2e7fcb687d6965a90fe3381774ebf23624fdc9ebf88336b"}
Jul 15 21:29:08.050625 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:08.050581    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k" event={"ID":"312dafc1-a9a9-44ae-9288-de82c7fb439f","Type":"ContainerStarted","Data":"1ba84374f6c990ded86acdd9461a3ce1c36f47d07a5f542363651c777b4434ee"}
Jul 15 21:29:08.080469 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:08.080419    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k" podStartSLOduration=1.080403374 podStartE2EDuration="1.080403374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:29:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:29:08.076143547 +0000 UTC m=+2481.225181604" watchObservedRunningTime="2026-07-15 21:29:08.080403374 +0000 UTC m=+2481.229441430"
Jul 15 21:29:08.412681 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:08.412593    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k"
Jul 15 21:29:08.417561 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:08.417537    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k"
Jul 15 21:29:09.055673 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:09.055642    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k"
Jul 15 21:29:09.056492 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:09.056474    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k"
Jul 15 21:29:10.160602 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:10.160562    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:29:10.261517 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:10.261473    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:29:10.262561 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:10.262527    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f5fcf1-766d7779b4-7fnfr"]
Jul 15 21:29:10.263006 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:10.262987    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd207757-2b22-4a73-838e-e0ed514ee510" containerName="istio-proxy"
Jul 15 21:29:10.263006 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:10.263005    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd207757-2b22-4a73-838e-e0ed514ee510" containerName="istio-proxy"
Jul 15 21:29:10.265164 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:10.265145    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5fcf1-766d7779b4-7fnfr"
Jul 15 21:29:10.269175 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:10.268509    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 15 21:29:10.269175 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:10.268568    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-f5fcf1-serving-cert\""
Jul 15 21:29:10.269175 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:10.268686    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 15 21:29:10.269175 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:10.269145    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-s2l9d\""
Jul 15 21:29:10.269653 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:10.269487    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f5fcf1-766d7779b4-7fnfr"]
Jul 15 21:29:10.310992 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:10.310964    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5twh\" (UniqueName: \"kubernetes.io/projected/57ddb433-beb7-4734-a0ba-24fd8106553e-kube-api-access-m5twh\") pod \"maas-api-e2e-api-a-f5fcf1-766d7779b4-7fnfr\" (UID: \"57ddb433-beb7-4734-a0ba-24fd8106553e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5fcf1-766d7779b4-7fnfr"
Jul 15 21:29:10.311130 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:10.311005    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57ddb433-beb7-4734-a0ba-24fd8106553e-maas-api-tls\") pod \"maas-api-e2e-api-a-f5fcf1-766d7779b4-7fnfr\" (UID: \"57ddb433-beb7-4734-a0ba-24fd8106553e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5fcf1-766d7779b4-7fnfr"
Jul 15 21:29:10.311130 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:10.311044    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57ddb433-beb7-4734-a0ba-24fd8106553e-openshift-service-ca\") pod \"maas-api-e2e-api-a-f5fcf1-766d7779b4-7fnfr\" (UID: \"57ddb433-beb7-4734-a0ba-24fd8106553e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5fcf1-766d7779b4-7fnfr"
Jul 15 21:29:10.331109 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:10.331079    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:29:10.412449 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:10.412358    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5twh\" (UniqueName: \"kubernetes.io/projected/57ddb433-beb7-4734-a0ba-24fd8106553e-kube-api-access-m5twh\") pod \"maas-api-e2e-api-a-f5fcf1-766d7779b4-7fnfr\" (UID: \"57ddb433-beb7-4734-a0ba-24fd8106553e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5fcf1-766d7779b4-7fnfr"
Jul 15 21:29:10.412449 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:10.412420    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57ddb433-beb7-4734-a0ba-24fd8106553e-maas-api-tls\") pod \"maas-api-e2e-api-a-f5fcf1-766d7779b4-7fnfr\" (UID: \"57ddb433-beb7-4734-a0ba-24fd8106553e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5fcf1-766d7779b4-7fnfr"
Jul 15 21:29:10.412688 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:10.412494    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57ddb433-beb7-4734-a0ba-24fd8106553e-openshift-service-ca\") pod \"maas-api-e2e-api-a-f5fcf1-766d7779b4-7fnfr\" (UID: \"57ddb433-beb7-4734-a0ba-24fd8106553e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5fcf1-766d7779b4-7fnfr"
Jul 15 21:29:10.412688 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:29:10.412520    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-f5fcf1-serving-cert: secret "maas-api-e2e-api-a-f5fcf1-serving-cert" not found
Jul 15 21:29:10.412688 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:29:10.412586    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57ddb433-beb7-4734-a0ba-24fd8106553e-maas-api-tls podName:57ddb433-beb7-4734-a0ba-24fd8106553e nodeName:}" failed. No retries permitted until 2026-07-15 21:29:10.912568104 +0000 UTC m=+2484.061606145 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/57ddb433-beb7-4734-a0ba-24fd8106553e-maas-api-tls") pod "maas-api-e2e-api-a-f5fcf1-766d7779b4-7fnfr" (UID: "57ddb433-beb7-4734-a0ba-24fd8106553e") : secret "maas-api-e2e-api-a-f5fcf1-serving-cert" not found
Jul 15 21:29:10.413221 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:10.413198    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57ddb433-beb7-4734-a0ba-24fd8106553e-openshift-service-ca\") pod \"maas-api-e2e-api-a-f5fcf1-766d7779b4-7fnfr\" (UID: \"57ddb433-beb7-4734-a0ba-24fd8106553e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5fcf1-766d7779b4-7fnfr"
Jul 15 21:29:10.424860 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:10.424838    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5twh\" (UniqueName: \"kubernetes.io/projected/57ddb433-beb7-4734-a0ba-24fd8106553e-kube-api-access-m5twh\") pod \"maas-api-e2e-api-a-f5fcf1-766d7779b4-7fnfr\" (UID: \"57ddb433-beb7-4734-a0ba-24fd8106553e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5fcf1-766d7779b4-7fnfr"
Jul 15 21:29:10.508187 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:10.508158    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:29:10.882742 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:10.882696    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:29:10.917622 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:10.917582    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57ddb433-beb7-4734-a0ba-24fd8106553e-maas-api-tls\") pod \"maas-api-e2e-api-a-f5fcf1-766d7779b4-7fnfr\" (UID: \"57ddb433-beb7-4734-a0ba-24fd8106553e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5fcf1-766d7779b4-7fnfr"
Jul 15 21:29:10.920024 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:10.920002    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57ddb433-beb7-4734-a0ba-24fd8106553e-maas-api-tls\") pod \"maas-api-e2e-api-a-f5fcf1-766d7779b4-7fnfr\" (UID: \"57ddb433-beb7-4734-a0ba-24fd8106553e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5fcf1-766d7779b4-7fnfr"
Jul 15 21:29:10.982819 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:10.982783    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:29:11.179344 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:11.179307    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5fcf1-766d7779b4-7fnfr"
Jul 15 21:29:11.299051 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:11.299019    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877"
Jul 15 21:29:11.322372 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:11.322330    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a27f0da-6504-4248-bd38-1f272f8c05a4-workload-socket\") pod \"3a27f0da-6504-4248-bd38-1f272f8c05a4\" (UID: \"3a27f0da-6504-4248-bd38-1f272f8c05a4\") "
Jul 15 21:29:11.322540 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:11.322405    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a27f0da-6504-4248-bd38-1f272f8c05a4-istio-data\") pod \"3a27f0da-6504-4248-bd38-1f272f8c05a4\" (UID: \"3a27f0da-6504-4248-bd38-1f272f8c05a4\") "
Jul 15 21:29:11.322540 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:11.322435    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a27f0da-6504-4248-bd38-1f272f8c05a4-workload-certs\") pod \"3a27f0da-6504-4248-bd38-1f272f8c05a4\" (UID: \"3a27f0da-6504-4248-bd38-1f272f8c05a4\") "
Jul 15 21:29:11.322540 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:11.322463    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a27f0da-6504-4248-bd38-1f272f8c05a4-credential-socket\") pod \"3a27f0da-6504-4248-bd38-1f272f8c05a4\" (UID: \"3a27f0da-6504-4248-bd38-1f272f8c05a4\") "
Jul 15 21:29:11.322540 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:11.322492    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a27f0da-6504-4248-bd38-1f272f8c05a4-istiod-ca-cert\") pod \"3a27f0da-6504-4248-bd38-1f272f8c05a4\" (UID: \"3a27f0da-6504-4248-bd38-1f272f8c05a4\") "
Jul 15 21:29:11.322540 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:11.322529    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a27f0da-6504-4248-bd38-1f272f8c05a4-istio-envoy\") pod \"3a27f0da-6504-4248-bd38-1f272f8c05a4\" (UID: \"3a27f0da-6504-4248-bd38-1f272f8c05a4\") "
Jul 15 21:29:11.322821 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:11.322559    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a27f0da-6504-4248-bd38-1f272f8c05a4-istio-token\") pod \"3a27f0da-6504-4248-bd38-1f272f8c05a4\" (UID: \"3a27f0da-6504-4248-bd38-1f272f8c05a4\") "
Jul 15 21:29:11.322821 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:11.322590    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mfwlf\" (UniqueName: \"kubernetes.io/projected/3a27f0da-6504-4248-bd38-1f272f8c05a4-kube-api-access-mfwlf\") pod \"3a27f0da-6504-4248-bd38-1f272f8c05a4\" (UID: \"3a27f0da-6504-4248-bd38-1f272f8c05a4\") "
Jul 15 21:29:11.322821 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:11.322585    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a27f0da-6504-4248-bd38-1f272f8c05a4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3a27f0da-6504-4248-bd38-1f272f8c05a4" (UID: "3a27f0da-6504-4248-bd38-1f272f8c05a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:29:11.322821 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:11.322627    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a27f0da-6504-4248-bd38-1f272f8c05a4-istio-podinfo\") pod \"3a27f0da-6504-4248-bd38-1f272f8c05a4\" (UID: \"3a27f0da-6504-4248-bd38-1f272f8c05a4\") "
Jul 15 21:29:11.323050 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:11.322843    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a27f0da-6504-4248-bd38-1f272f8c05a4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3a27f0da-6504-4248-bd38-1f272f8c05a4" (UID: "3a27f0da-6504-4248-bd38-1f272f8c05a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 21:29:11.323050 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:11.322981    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a27f0da-6504-4248-bd38-1f272f8c05a4-istiod-ca-cert\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:29:11.323050 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:11.322998    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a27f0da-6504-4248-bd38-1f272f8c05a4-workload-socket\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:29:11.323211 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:11.323146    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a27f0da-6504-4248-bd38-1f272f8c05a4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3a27f0da-6504-4248-bd38-1f272f8c05a4" (UID: "3a27f0da-6504-4248-bd38-1f272f8c05a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:29:11.323315 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:11.323292    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a27f0da-6504-4248-bd38-1f272f8c05a4-istio-data" (OuterVolumeSpecName: "istio-data") pod "3a27f0da-6504-4248-bd38-1f272f8c05a4" (UID: "3a27f0da-6504-4248-bd38-1f272f8c05a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:29:11.323376 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:11.323352    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a27f0da-6504-4248-bd38-1f272f8c05a4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3a27f0da-6504-4248-bd38-1f272f8c05a4" (UID: "3a27f0da-6504-4248-bd38-1f272f8c05a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:29:11.325114 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:11.325087    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a27f0da-6504-4248-bd38-1f272f8c05a4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3a27f0da-6504-4248-bd38-1f272f8c05a4" (UID: "3a27f0da-6504-4248-bd38-1f272f8c05a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:29:11.325394 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:11.325367    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a27f0da-6504-4248-bd38-1f272f8c05a4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3a27f0da-6504-4248-bd38-1f272f8c05a4" (UID: "3a27f0da-6504-4248-bd38-1f272f8c05a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 15 21:29:11.325692 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:11.325670    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a27f0da-6504-4248-bd38-1f272f8c05a4-istio-token" (OuterVolumeSpecName: "istio-token") pod "3a27f0da-6504-4248-bd38-1f272f8c05a4" (UID: "3a27f0da-6504-4248-bd38-1f272f8c05a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 21:29:11.325752 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:11.325703    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a27f0da-6504-4248-bd38-1f272f8c05a4-kube-api-access-mfwlf" (OuterVolumeSpecName: "kube-api-access-mfwlf") pod "3a27f0da-6504-4248-bd38-1f272f8c05a4" (UID: "3a27f0da-6504-4248-bd38-1f272f8c05a4"). InnerVolumeSpecName "kube-api-access-mfwlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 21:29:11.347153 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:11.347120    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f5fcf1-766d7779b4-7fnfr"]
Jul 15 21:29:11.348659 ip-10-0-137-234 kubenswrapper[2566]: W0715 21:29:11.348628    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57ddb433_beb7_4734_a0ba_24fd8106553e.slice/crio-fe57e29cc33ae4b9055f1f188e4eb6bc1b013e05db036bbcc512c0e7809c35a5 WatchSource:0}: Error finding container fe57e29cc33ae4b9055f1f188e4eb6bc1b013e05db036bbcc512c0e7809c35a5: Status 404 returned error can't find the container with id fe57e29cc33ae4b9055f1f188e4eb6bc1b013e05db036bbcc512c0e7809c35a5
Jul 15 21:29:11.350352 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:11.350328    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 15 21:29:11.424467 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:11.424395    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a27f0da-6504-4248-bd38-1f272f8c05a4-istio-data\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:29:11.424467 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:11.424422    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a27f0da-6504-4248-bd38-1f272f8c05a4-workload-certs\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:29:11.424467 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:11.424432    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a27f0da-6504-4248-bd38-1f272f8c05a4-credential-socket\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:29:11.424467 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:11.424441    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a27f0da-6504-4248-bd38-1f272f8c05a4-istio-envoy\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:29:11.424467 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:11.424449    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a27f0da-6504-4248-bd38-1f272f8c05a4-istio-token\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:29:11.424467 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:11.424457    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfwlf\" (UniqueName: \"kubernetes.io/projected/3a27f0da-6504-4248-bd38-1f272f8c05a4-kube-api-access-mfwlf\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:29:11.424467 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:11.424465    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a27f0da-6504-4248-bd38-1f272f8c05a4-istio-podinfo\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:29:12.071009 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:12.070922    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5fcf1-766d7779b4-7fnfr" event={"ID":"57ddb433-beb7-4734-a0ba-24fd8106553e","Type":"ContainerStarted","Data":"fe57e29cc33ae4b9055f1f188e4eb6bc1b013e05db036bbcc512c0e7809c35a5"}
Jul 15 21:29:12.072814 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:12.072782    2566 generic.go:356] "Generic (PLEG): container finished" podID="3a27f0da-6504-4248-bd38-1f272f8c05a4" containerID="185b33f6c9a6c7cac5a2a53a3f16d33118aa81e7f9f038e931bde11d259604b1" exitCode=0
Jul 15 21:29:12.072971 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:12.072832    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877" event={"ID":"3a27f0da-6504-4248-bd38-1f272f8c05a4","Type":"ContainerDied","Data":"185b33f6c9a6c7cac5a2a53a3f16d33118aa81e7f9f038e931bde11d259604b1"}
Jul 15 21:29:12.072971 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:12.072874    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877" event={"ID":"3a27f0da-6504-4248-bd38-1f272f8c05a4","Type":"ContainerDied","Data":"9de9277b6afb3638643fce05d2e06d01a250e6ae9c2f150dc88552e071b4ecef"}
Jul 15 21:29:12.072971 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:12.072836    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877"
Jul 15 21:29:12.072971 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:12.072931    2566 scope.go:117] "RemoveContainer" containerID="185b33f6c9a6c7cac5a2a53a3f16d33118aa81e7f9f038e931bde11d259604b1"
Jul 15 21:29:12.082767 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:12.082743    2566 scope.go:117] "RemoveContainer" containerID="3df075f0e1a9a010bf5d3a332b71ddfe10ce3c1625fae86e7096dd094c5778c4"
Jul 15 21:29:12.091728 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:12.091708    2566 scope.go:117] "RemoveContainer" containerID="185b33f6c9a6c7cac5a2a53a3f16d33118aa81e7f9f038e931bde11d259604b1"
Jul 15 21:29:12.092094 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:29:12.092070    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"185b33f6c9a6c7cac5a2a53a3f16d33118aa81e7f9f038e931bde11d259604b1\": container with ID starting with 185b33f6c9a6c7cac5a2a53a3f16d33118aa81e7f9f038e931bde11d259604b1 not found: ID does not exist" containerID="185b33f6c9a6c7cac5a2a53a3f16d33118aa81e7f9f038e931bde11d259604b1"
Jul 15 21:29:12.092160 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:12.092102    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"185b33f6c9a6c7cac5a2a53a3f16d33118aa81e7f9f038e931bde11d259604b1"} err="failed to get container status \"185b33f6c9a6c7cac5a2a53a3f16d33118aa81e7f9f038e931bde11d259604b1\": rpc error: code = NotFound desc = could not find container \"185b33f6c9a6c7cac5a2a53a3f16d33118aa81e7f9f038e931bde11d259604b1\": container with ID starting with 185b33f6c9a6c7cac5a2a53a3f16d33118aa81e7f9f038e931bde11d259604b1 not found: ID does not exist"
Jul 15 21:29:12.092160 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:12.092122    2566 scope.go:117] "RemoveContainer" containerID="3df075f0e1a9a010bf5d3a332b71ddfe10ce3c1625fae86e7096dd094c5778c4"
Jul 15 21:29:12.092368 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:29:12.092349    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3df075f0e1a9a010bf5d3a332b71ddfe10ce3c1625fae86e7096dd094c5778c4\": container with ID starting with 3df075f0e1a9a010bf5d3a332b71ddfe10ce3c1625fae86e7096dd094c5778c4 not found: ID does not exist" containerID="3df075f0e1a9a010bf5d3a332b71ddfe10ce3c1625fae86e7096dd094c5778c4"
Jul 15 21:29:12.092405 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:12.092376    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3df075f0e1a9a010bf5d3a332b71ddfe10ce3c1625fae86e7096dd094c5778c4"} err="failed to get container status \"3df075f0e1a9a010bf5d3a332b71ddfe10ce3c1625fae86e7096dd094c5778c4\": rpc error: code = NotFound desc = could not find container \"3df075f0e1a9a010bf5d3a332b71ddfe10ce3c1625fae86e7096dd094c5778c4\": container with ID starting with 3df075f0e1a9a010bf5d3a332b71ddfe10ce3c1625fae86e7096dd094c5778c4 not found: ID does not exist"
Jul 15 21:29:12.099684 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:12.099661    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877"]
Jul 15 21:29:12.104163 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:12.104143    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-659e77-openshift-default-7d6c97f7-p4877"]
Jul 15 21:29:13.080654 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:13.080619    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5fcf1-766d7779b4-7fnfr" event={"ID":"57ddb433-beb7-4734-a0ba-24fd8106553e","Type":"ContainerStarted","Data":"3f0d72310abe0e9e85de2f5634edd56fc7d1f01cd197ff135d0426f54df9cbbc"}
Jul 15 21:29:13.081113 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:13.080760    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5fcf1-766d7779b4-7fnfr"
Jul 15 21:29:13.102681 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:13.102579    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5fcf1-766d7779b4-7fnfr" podStartSLOduration=1.61485346 podStartE2EDuration="3.102560221s" totalImagesPullingTime="1.487706761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:29:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:29:13.101808284 +0000 UTC m=+2486.250846366" watchObservedRunningTime="2026-07-15 21:29:13.102560221 +0000 UTC m=+2486.251598279"
Jul 15 21:29:13.756464 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:13.756431    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a27f0da-6504-4248-bd38-1f272f8c05a4" path="/var/lib/kubelet/pods/3a27f0da-6504-4248-bd38-1f272f8c05a4/volumes"
Jul 15 21:29:14.634171 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:14.634141    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:29:15.154021 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:15.153984    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:29:15.460697 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:15.460660    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh"]
Jul 15 21:29:15.461062 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:15.461046    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a27f0da-6504-4248-bd38-1f272f8c05a4" containerName="istio-proxy"
Jul 15 21:29:15.461062 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:15.461063    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a27f0da-6504-4248-bd38-1f272f8c05a4" containerName="istio-proxy"
Jul 15 21:29:15.461199 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:15.461084    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a27f0da-6504-4248-bd38-1f272f8c05a4" containerName="istio-proxy"
Jul 15 21:29:15.461199 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:15.461090    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a27f0da-6504-4248-bd38-1f272f8c05a4" containerName="istio-proxy"
Jul 15 21:29:15.461199 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:15.461156    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a27f0da-6504-4248-bd38-1f272f8c05a4" containerName="istio-proxy"
Jul 15 21:29:15.461199 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:15.461167    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a27f0da-6504-4248-bd38-1f272f8c05a4" containerName="istio-proxy"
Jul 15 21:29:15.463204 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:15.463183    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh"
Jul 15 21:29:15.466276 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:15.466257    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-843ced-openshift-default-dockercfg-222xx\""
Jul 15 21:29:15.481197 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:15.481166    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh"]
Jul 15 21:29:15.497853 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:15.497822    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:29:15.563326 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:15.563297    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/570f9db7-c9cd-42f6-b044-ce7069a322ae-istio-podinfo\") pod \"e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh\" (UID: \"570f9db7-c9cd-42f6-b044-ce7069a322ae\") " pod="openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh"
Jul 15 21:29:15.563480 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:15.563335    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/570f9db7-c9cd-42f6-b044-ce7069a322ae-istio-token\") pod \"e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh\" (UID: \"570f9db7-c9cd-42f6-b044-ce7069a322ae\") " pod="openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh"
Jul 15 21:29:15.563480 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:15.563388    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/570f9db7-c9cd-42f6-b044-ce7069a322ae-workload-socket\") pod \"e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh\" (UID: \"570f9db7-c9cd-42f6-b044-ce7069a322ae\") " pod="openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh"
Jul 15 21:29:15.563480 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:15.563431    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/570f9db7-c9cd-42f6-b044-ce7069a322ae-istiod-ca-cert\") pod \"e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh\" (UID: \"570f9db7-c9cd-42f6-b044-ce7069a322ae\") " pod="openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh"
Jul 15 21:29:15.563480 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:15.563475    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/570f9db7-c9cd-42f6-b044-ce7069a322ae-istio-data\") pod \"e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh\" (UID: \"570f9db7-c9cd-42f6-b044-ce7069a322ae\") " pod="openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh"
Jul 15 21:29:15.563684 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:15.563509    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/570f9db7-c9cd-42f6-b044-ce7069a322ae-workload-certs\") pod \"e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh\" (UID: \"570f9db7-c9cd-42f6-b044-ce7069a322ae\") " pod="openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh"
Jul 15 21:29:15.563684 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:15.563546    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/570f9db7-c9cd-42f6-b044-ce7069a322ae-istio-envoy\") pod \"e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh\" (UID: \"570f9db7-c9cd-42f6-b044-ce7069a322ae\") " pod="openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh"
Jul 15 21:29:15.563684 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:15.563644    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6x59\" (UniqueName: \"kubernetes.io/projected/570f9db7-c9cd-42f6-b044-ce7069a322ae-kube-api-access-r6x59\") pod \"e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh\" (UID: \"570f9db7-c9cd-42f6-b044-ce7069a322ae\") " pod="openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh"
Jul 15 21:29:15.563803 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:15.563686    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/570f9db7-c9cd-42f6-b044-ce7069a322ae-credential-socket\") pod \"e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh\" (UID: \"570f9db7-c9cd-42f6-b044-ce7069a322ae\") " pod="openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh"
Jul 15 21:29:15.664742 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:15.664703    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/570f9db7-c9cd-42f6-b044-ce7069a322ae-workload-socket\") pod \"e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh\" (UID: \"570f9db7-c9cd-42f6-b044-ce7069a322ae\") " pod="openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh"
Jul 15 21:29:15.664742 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:15.664745    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/570f9db7-c9cd-42f6-b044-ce7069a322ae-istiod-ca-cert\") pod \"e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh\" (UID: \"570f9db7-c9cd-42f6-b044-ce7069a322ae\") " pod="openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh"
Jul 15 21:29:15.665300 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:15.664785    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/570f9db7-c9cd-42f6-b044-ce7069a322ae-istio-data\") pod \"e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh\" (UID: \"570f9db7-c9cd-42f6-b044-ce7069a322ae\") " pod="openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh"
Jul 15 21:29:15.665300 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:15.664816    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/570f9db7-c9cd-42f6-b044-ce7069a322ae-workload-certs\") pod \"e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh\" (UID: \"570f9db7-c9cd-42f6-b044-ce7069a322ae\") " pod="openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh"
Jul 15 21:29:15.665300 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:15.664849    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/570f9db7-c9cd-42f6-b044-ce7069a322ae-istio-envoy\") pod \"e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh\" (UID: \"570f9db7-c9cd-42f6-b044-ce7069a322ae\") " pod="openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh"
Jul 15 21:29:15.665300 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:15.664906    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6x59\" (UniqueName: \"kubernetes.io/projected/570f9db7-c9cd-42f6-b044-ce7069a322ae-kube-api-access-r6x59\") pod \"e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh\" (UID: \"570f9db7-c9cd-42f6-b044-ce7069a322ae\") " pod="openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh"
Jul 15 21:29:15.665300 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:15.664933    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/570f9db7-c9cd-42f6-b044-ce7069a322ae-credential-socket\") pod \"e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh\" (UID: \"570f9db7-c9cd-42f6-b044-ce7069a322ae\") " pod="openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh"
Jul 15 21:29:15.665300 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:15.664976    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/570f9db7-c9cd-42f6-b044-ce7069a322ae-istio-podinfo\") pod \"e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh\" (UID: \"570f9db7-c9cd-42f6-b044-ce7069a322ae\") " pod="openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh"
Jul 15 21:29:15.665300 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:15.665000    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/570f9db7-c9cd-42f6-b044-ce7069a322ae-istio-token\") pod \"e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh\" (UID: \"570f9db7-c9cd-42f6-b044-ce7069a322ae\") " pod="openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh"
Jul 15 21:29:15.665300 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:15.665177    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/570f9db7-c9cd-42f6-b044-ce7069a322ae-workload-socket\") pod \"e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh\" (UID: \"570f9db7-c9cd-42f6-b044-ce7069a322ae\") " pod="openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh"
Jul 15 21:29:15.665659 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:15.665344    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/570f9db7-c9cd-42f6-b044-ce7069a322ae-workload-certs\") pod \"e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh\" (UID: \"570f9db7-c9cd-42f6-b044-ce7069a322ae\") " pod="openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh"
Jul 15 21:29:15.665659 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:15.665394    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/570f9db7-c9cd-42f6-b044-ce7069a322ae-istio-data\") pod \"e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh\" (UID: \"570f9db7-c9cd-42f6-b044-ce7069a322ae\") " pod="openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh"
Jul 15 21:29:15.665659 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:15.665491    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/570f9db7-c9cd-42f6-b044-ce7069a322ae-credential-socket\") pod \"e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh\" (UID: \"570f9db7-c9cd-42f6-b044-ce7069a322ae\") " pod="openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh"
Jul 15 21:29:15.665659 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:15.665606    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/570f9db7-c9cd-42f6-b044-ce7069a322ae-istiod-ca-cert\") pod \"e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh\" (UID: \"570f9db7-c9cd-42f6-b044-ce7069a322ae\") " pod="openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh"
Jul 15 21:29:15.667357 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:15.667334    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/570f9db7-c9cd-42f6-b044-ce7069a322ae-istio-envoy\") pod \"e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh\" (UID: \"570f9db7-c9cd-42f6-b044-ce7069a322ae\") " pod="openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh"
Jul 15 21:29:15.667573 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:15.667556    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/570f9db7-c9cd-42f6-b044-ce7069a322ae-istio-podinfo\") pod \"e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh\" (UID: \"570f9db7-c9cd-42f6-b044-ce7069a322ae\") " pod="openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh"
Jul 15 21:29:15.677884 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:15.677849    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6x59\" (UniqueName: \"kubernetes.io/projected/570f9db7-c9cd-42f6-b044-ce7069a322ae-kube-api-access-r6x59\") pod \"e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh\" (UID: \"570f9db7-c9cd-42f6-b044-ce7069a322ae\") " pod="openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh"
Jul 15 21:29:15.687511 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:15.687485    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/570f9db7-c9cd-42f6-b044-ce7069a322ae-istio-token\") pod \"e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh\" (UID: \"570f9db7-c9cd-42f6-b044-ce7069a322ae\") " pod="openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh"
Jul 15 21:29:15.768363 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:15.768283    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:29:15.773262 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:15.773238    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh"
Jul 15 21:29:15.922429 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:15.922397    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh"]
Jul 15 21:29:15.924197 ip-10-0-137-234 kubenswrapper[2566]: W0715 21:29:15.924169    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod570f9db7_c9cd_42f6_b044_ce7069a322ae.slice/crio-9772dcc326b72bed5a8e59e0d266d5614d038171a1a743021de0d93fb47e2aef WatchSource:0}: Error finding container 9772dcc326b72bed5a8e59e0d266d5614d038171a1a743021de0d93fb47e2aef: Status 404 returned error can't find the container with id 9772dcc326b72bed5a8e59e0d266d5614d038171a1a743021de0d93fb47e2aef
Jul 15 21:29:15.926278 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:15.926251    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:29:15.926349 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:15.926318    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:29:15.926393 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:15.926347    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:29:16.098185 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:16.098101    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh" event={"ID":"570f9db7-c9cd-42f6-b044-ce7069a322ae","Type":"ContainerStarted","Data":"f6110fecd01c2446d3ca47bd5a56f4a06ec76d014bfee1a74c7e609969295625"}
Jul 15 21:29:16.098185 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:16.098160    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh" event={"ID":"570f9db7-c9cd-42f6-b044-ce7069a322ae","Type":"ContainerStarted","Data":"9772dcc326b72bed5a8e59e0d266d5614d038171a1a743021de0d93fb47e2aef"}
Jul 15 21:29:16.127405 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:16.127345    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh" podStartSLOduration=1.127310788 podStartE2EDuration="1.127310788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:29:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:29:16.125568747 +0000 UTC m=+2489.274606805" watchObservedRunningTime="2026-07-15 21:29:16.127310788 +0000 UTC m=+2489.276348845"
Jul 15 21:29:16.773444 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:16.773399    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh"
Jul 15 21:29:16.787917 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:16.787196    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh"
Jul 15 21:29:17.105511 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:17.105410    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh"
Jul 15 21:29:17.106536 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:17.106511    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh"
Jul 15 21:29:18.165273 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:18.165235    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:29:18.261025 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:18.260989    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:29:18.279997 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:18.279957    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-843ced-fb5f4b8cf-zqpsf"]
Jul 15 21:29:18.283323 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:18.283299    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-843ced-fb5f4b8cf-zqpsf"
Jul 15 21:29:18.289427 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:18.289404    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-843ced-serving-cert\""
Jul 15 21:29:18.297671 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:18.297637    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-843ced-fb5f4b8cf-zqpsf"]
Jul 15 21:29:18.360960 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:18.360927    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:29:18.407405 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:18.407358    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dae6215-3e23-49a1-879d-26881faae017-maas-api-tls\") pod \"maas-api-e2e-api-b-843ced-fb5f4b8cf-zqpsf\" (UID: \"4dae6215-3e23-49a1-879d-26881faae017\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-843ced-fb5f4b8cf-zqpsf"
Jul 15 21:29:18.407604 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:18.407452    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nftd6\" (UniqueName: \"kubernetes.io/projected/4dae6215-3e23-49a1-879d-26881faae017-kube-api-access-nftd6\") pod \"maas-api-e2e-api-b-843ced-fb5f4b8cf-zqpsf\" (UID: \"4dae6215-3e23-49a1-879d-26881faae017\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-843ced-fb5f4b8cf-zqpsf"
Jul 15 21:29:18.407604 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:18.407485    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dae6215-3e23-49a1-879d-26881faae017-openshift-service-ca\") pod \"maas-api-e2e-api-b-843ced-fb5f4b8cf-zqpsf\" (UID: \"4dae6215-3e23-49a1-879d-26881faae017\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-843ced-fb5f4b8cf-zqpsf"
Jul 15 21:29:18.508415 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:18.508373    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dae6215-3e23-49a1-879d-26881faae017-maas-api-tls\") pod \"maas-api-e2e-api-b-843ced-fb5f4b8cf-zqpsf\" (UID: \"4dae6215-3e23-49a1-879d-26881faae017\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-843ced-fb5f4b8cf-zqpsf"
Jul 15 21:29:18.508619 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:18.508432    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nftd6\" (UniqueName: \"kubernetes.io/projected/4dae6215-3e23-49a1-879d-26881faae017-kube-api-access-nftd6\") pod \"maas-api-e2e-api-b-843ced-fb5f4b8cf-zqpsf\" (UID: \"4dae6215-3e23-49a1-879d-26881faae017\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-843ced-fb5f4b8cf-zqpsf"
Jul 15 21:29:18.508619 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:18.508452    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dae6215-3e23-49a1-879d-26881faae017-openshift-service-ca\") pod \"maas-api-e2e-api-b-843ced-fb5f4b8cf-zqpsf\" (UID: \"4dae6215-3e23-49a1-879d-26881faae017\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-843ced-fb5f4b8cf-zqpsf"
Jul 15 21:29:18.508619 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:29:18.508606    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-843ced-serving-cert: secret "maas-api-e2e-api-b-843ced-serving-cert" not found
Jul 15 21:29:18.508797 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:29:18.508688    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dae6215-3e23-49a1-879d-26881faae017-maas-api-tls podName:4dae6215-3e23-49a1-879d-26881faae017 nodeName:}" failed. No retries permitted until 2026-07-15 21:29:19.008664968 +0000 UTC m=+2492.157703016 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4dae6215-3e23-49a1-879d-26881faae017-maas-api-tls") pod "maas-api-e2e-api-b-843ced-fb5f4b8cf-zqpsf" (UID: "4dae6215-3e23-49a1-879d-26881faae017") : secret "maas-api-e2e-api-b-843ced-serving-cert" not found
Jul 15 21:29:18.509190 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:18.509170    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dae6215-3e23-49a1-879d-26881faae017-openshift-service-ca\") pod \"maas-api-e2e-api-b-843ced-fb5f4b8cf-zqpsf\" (UID: \"4dae6215-3e23-49a1-879d-26881faae017\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-843ced-fb5f4b8cf-zqpsf"
Jul 15 21:29:18.525507 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:18.525482    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nftd6\" (UniqueName: \"kubernetes.io/projected/4dae6215-3e23-49a1-879d-26881faae017-kube-api-access-nftd6\") pod \"maas-api-e2e-api-b-843ced-fb5f4b8cf-zqpsf\" (UID: \"4dae6215-3e23-49a1-879d-26881faae017\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-843ced-fb5f4b8cf-zqpsf"
Jul 15 21:29:18.623417 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:18.623383    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:29:19.013813 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:19.013778    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dae6215-3e23-49a1-879d-26881faae017-maas-api-tls\") pod \"maas-api-e2e-api-b-843ced-fb5f4b8cf-zqpsf\" (UID: \"4dae6215-3e23-49a1-879d-26881faae017\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-843ced-fb5f4b8cf-zqpsf"
Jul 15 21:29:19.016341 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:19.016305    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dae6215-3e23-49a1-879d-26881faae017-maas-api-tls\") pod \"maas-api-e2e-api-b-843ced-fb5f4b8cf-zqpsf\" (UID: \"4dae6215-3e23-49a1-879d-26881faae017\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-843ced-fb5f4b8cf-zqpsf"
Jul 15 21:29:19.095535 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:19.094643    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5fcf1-766d7779b4-7fnfr"
Jul 15 21:29:19.162504 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:19.162467    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:29:19.219793 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:19.219764    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-843ced-fb5f4b8cf-zqpsf"
Jul 15 21:29:19.263471 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:19.263437    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:29:19.605315 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:19.596829    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-843ced-fb5f4b8cf-zqpsf"]
Jul 15 21:29:19.606560 ip-10-0-137-234 kubenswrapper[2566]: W0715 21:29:19.606511    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dae6215_3e23_49a1_879d_26881faae017.slice/crio-1a6b198e29dc4da096fd4f5af328a16bcec6037d5913354c7c378305a692365d WatchSource:0}: Error finding container 1a6b198e29dc4da096fd4f5af328a16bcec6037d5913354c7c378305a692365d: Status 404 returned error can't find the container with id 1a6b198e29dc4da096fd4f5af328a16bcec6037d5913354c7c378305a692365d
Jul 15 21:29:20.122736 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:20.122702    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-843ced-fb5f4b8cf-zqpsf" event={"ID":"4dae6215-3e23-49a1-879d-26881faae017","Type":"ContainerStarted","Data":"d30ab4b793e85148af19b501e1d60e866b2d842bb207be6d81bf748660f04ee0"}
Jul 15 21:29:20.122947 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:20.122759    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-843ced-fb5f4b8cf-zqpsf" event={"ID":"4dae6215-3e23-49a1-879d-26881faae017","Type":"ContainerStarted","Data":"1a6b198e29dc4da096fd4f5af328a16bcec6037d5913354c7c378305a692365d"}
Jul 15 21:29:20.122947 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:20.122801    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-843ced-fb5f4b8cf-zqpsf"
Jul 15 21:29:20.143325 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:20.143259    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-843ced-fb5f4b8cf-zqpsf" podStartSLOduration=1.844956365 podStartE2EDuration="2.143242833s" totalImagesPullingTime="298.286468ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:29:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:29:20.142906286 +0000 UTC m=+2493.291944336" watchObservedRunningTime="2026-07-15 21:29:20.143242833 +0000 UTC m=+2493.292280890"
Jul 15 21:29:22.718588 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:22.718558    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:29:24.056911 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:24.056872    2566 patch_prober.go:28] interesting pod/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Jul 15 21:29:24.057319 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:24.056948    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k" podUID="312dafc1-a9a9-44ae-9288-de82c7fb439f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Jul 15 21:29:24.146915 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:24.146867    2566 generic.go:356] "Generic (PLEG): container finished" podID="312dafc1-a9a9-44ae-9288-de82c7fb439f" containerID="514b3d940dd98131d2e7fcb687d6965a90fe3381774ebf23624fdc9ebf88336b" exitCode=0
Jul 15 21:29:24.147076 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:24.146929    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k" event={"ID":"312dafc1-a9a9-44ae-9288-de82c7fb439f","Type":"ContainerDied","Data":"514b3d940dd98131d2e7fcb687d6965a90fe3381774ebf23624fdc9ebf88336b"}
Jul 15 21:29:24.147332 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:24.147318    2566 scope.go:117] "RemoveContainer" containerID="514b3d940dd98131d2e7fcb687d6965a90fe3381774ebf23624fdc9ebf88336b"
Jul 15 21:29:24.148304 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:24.148280    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:29:24.148357 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:24.148346    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:29:24.148392 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:24.148374    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:29:25.154746 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:25.154710    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k" event={"ID":"312dafc1-a9a9-44ae-9288-de82c7fb439f","Type":"ContainerStarted","Data":"24c19f4123d68898adf282bceeb58eb9111d2415624589a277ff227d70517b26"}
Jul 15 21:29:25.412934 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:25.412812    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k"
Jul 15 21:29:25.413346 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:25.413219    2566 patch_prober.go:28] interesting pod/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Jul 15 21:29:25.413566 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:25.413541    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k" podUID="312dafc1-a9a9-44ae-9288-de82c7fb439f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Jul 15 21:29:26.135697 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:26.135662    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-843ced-fb5f4b8cf-zqpsf"
Jul 15 21:29:26.413216 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:26.413101    2566 patch_prober.go:28] interesting pod/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Jul 15 21:29:26.413216 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:26.413192    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k" podUID="312dafc1-a9a9-44ae-9288-de82c7fb439f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Jul 15 21:29:27.413143 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:27.413096    2566 patch_prober.go:28] interesting pod/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Jul 15 21:29:27.413377 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:27.413170    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k" podUID="312dafc1-a9a9-44ae-9288-de82c7fb439f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Jul 15 21:29:28.412347 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:28.412308    2566 patch_prober.go:28] interesting pod/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Jul 15 21:29:28.412519 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:28.412378    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k" podUID="312dafc1-a9a9-44ae-9288-de82c7fb439f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Jul 15 21:29:29.416971 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:29.416943    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k"
Jul 15 21:29:29.417361 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:29.417338    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k"
Jul 15 21:29:29.418243 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:29.418224    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k"
Jul 15 21:29:31.188742 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:31.188643    2566 generic.go:356] "Generic (PLEG): container finished" podID="570f9db7-c9cd-42f6-b044-ce7069a322ae" containerID="f6110fecd01c2446d3ca47bd5a56f4a06ec76d014bfee1a74c7e609969295625" exitCode=0
Jul 15 21:29:31.188742 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:31.188722    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh" event={"ID":"570f9db7-c9cd-42f6-b044-ce7069a322ae","Type":"ContainerDied","Data":"f6110fecd01c2446d3ca47bd5a56f4a06ec76d014bfee1a74c7e609969295625"}
Jul 15 21:29:31.189224 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:31.189206    2566 scope.go:117] "RemoveContainer" containerID="f6110fecd01c2446d3ca47bd5a56f4a06ec76d014bfee1a74c7e609969295625"
Jul 15 21:29:31.190215 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:31.190192    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:29:31.190275 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:31.190263    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:29:31.190311 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:31.190297    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:29:31.705658 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:31.705620    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:29:31.804495 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:31.804458    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:29:32.105831 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:32.105735    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh"
Jul 15 21:29:32.206821 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:32.206783    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh" event={"ID":"570f9db7-c9cd-42f6-b044-ce7069a322ae","Type":"ContainerStarted","Data":"a20c154e0d5e32c9d4df5a908a02052d6f20f76a90ef6f76e548f5afe729a404"}
Jul 15 21:29:32.774031 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:32.773994    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh"
Jul 15 21:29:32.775105 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:32.775078    2566 patch_prober.go:28] interesting pod/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Jul 15 21:29:32.775235 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:32.775129    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh" podUID="570f9db7-c9cd-42f6-b044-ce7069a322ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Jul 15 21:29:33.773732 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:33.773692    2566 patch_prober.go:28] interesting pod/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Jul 15 21:29:33.774231 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:33.773754    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh" podUID="570f9db7-c9cd-42f6-b044-ce7069a322ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Jul 15 21:29:34.774137 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:34.774104    2566 patch_prober.go:28] interesting pod/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Jul 15 21:29:34.774570 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:34.774162    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh" podUID="570f9db7-c9cd-42f6-b044-ce7069a322ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Jul 15 21:29:35.773740 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:35.773701    2566 patch_prober.go:28] interesting pod/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Jul 15 21:29:35.773971 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:35.773775    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh" podUID="570f9db7-c9cd-42f6-b044-ce7069a322ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Jul 15 21:29:36.777676 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:36.777648    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh"
Jul 15 21:29:36.778138 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:36.778121    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh"
Jul 15 21:29:36.778828 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:36.778805    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh"
Jul 15 21:29:40.114043 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:40.113960    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:29:40.214169 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:40.214134    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:29:41.561583 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:41.561542    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:29:41.664101 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:41.664047    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:29:41.833496 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:41.833416    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f5fcf1-766d7779b4-7fnfr"]
Jul 15 21:29:41.833685 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:41.833660    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5fcf1-766d7779b4-7fnfr" podUID="57ddb433-beb7-4734-a0ba-24fd8106553e" containerName="maas-api" containerID="cri-o://3f0d72310abe0e9e85de2f5634edd56fc7d1f01cd197ff135d0426f54df9cbbc" gracePeriod=30
Jul 15 21:29:41.864335 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:41.864311    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:29:42.093999 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:42.093924    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5fcf1-766d7779b4-7fnfr"
Jul 15 21:29:42.235120 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:42.235086    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57ddb433-beb7-4734-a0ba-24fd8106553e-maas-api-tls\") pod \"57ddb433-beb7-4734-a0ba-24fd8106553e\" (UID: \"57ddb433-beb7-4734-a0ba-24fd8106553e\") "
Jul 15 21:29:42.235318 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:42.235142    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57ddb433-beb7-4734-a0ba-24fd8106553e-openshift-service-ca\") pod \"57ddb433-beb7-4734-a0ba-24fd8106553e\" (UID: \"57ddb433-beb7-4734-a0ba-24fd8106553e\") "
Jul 15 21:29:42.235318 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:42.235213    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5twh\" (UniqueName: \"kubernetes.io/projected/57ddb433-beb7-4734-a0ba-24fd8106553e-kube-api-access-m5twh\") pod \"57ddb433-beb7-4734-a0ba-24fd8106553e\" (UID: \"57ddb433-beb7-4734-a0ba-24fd8106553e\") "
Jul 15 21:29:42.235567 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:42.235543    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57ddb433-beb7-4734-a0ba-24fd8106553e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "57ddb433-beb7-4734-a0ba-24fd8106553e" (UID: "57ddb433-beb7-4734-a0ba-24fd8106553e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 21:29:42.237221 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:42.237193    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57ddb433-beb7-4734-a0ba-24fd8106553e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "57ddb433-beb7-4734-a0ba-24fd8106553e" (UID: "57ddb433-beb7-4734-a0ba-24fd8106553e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 21:29:42.237336 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:42.237256    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57ddb433-beb7-4734-a0ba-24fd8106553e-kube-api-access-m5twh" (OuterVolumeSpecName: "kube-api-access-m5twh") pod "57ddb433-beb7-4734-a0ba-24fd8106553e" (UID: "57ddb433-beb7-4734-a0ba-24fd8106553e"). InnerVolumeSpecName "kube-api-access-m5twh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 21:29:42.258622 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:42.258592    2566 generic.go:356] "Generic (PLEG): container finished" podID="57ddb433-beb7-4734-a0ba-24fd8106553e" containerID="3f0d72310abe0e9e85de2f5634edd56fc7d1f01cd197ff135d0426f54df9cbbc" exitCode=0
Jul 15 21:29:42.258742 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:42.258644    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5fcf1-766d7779b4-7fnfr"
Jul 15 21:29:42.258742 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:42.258672    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5fcf1-766d7779b4-7fnfr" event={"ID":"57ddb433-beb7-4734-a0ba-24fd8106553e","Type":"ContainerDied","Data":"3f0d72310abe0e9e85de2f5634edd56fc7d1f01cd197ff135d0426f54df9cbbc"}
Jul 15 21:29:42.258742 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:42.258724    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f5fcf1-766d7779b4-7fnfr" event={"ID":"57ddb433-beb7-4734-a0ba-24fd8106553e","Type":"ContainerDied","Data":"fe57e29cc33ae4b9055f1f188e4eb6bc1b013e05db036bbcc512c0e7809c35a5"}
Jul 15 21:29:42.258930 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:42.258746    2566 scope.go:117] "RemoveContainer" containerID="3f0d72310abe0e9e85de2f5634edd56fc7d1f01cd197ff135d0426f54df9cbbc"
Jul 15 21:29:42.267934 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:42.267916    2566 scope.go:117] "RemoveContainer" containerID="3f0d72310abe0e9e85de2f5634edd56fc7d1f01cd197ff135d0426f54df9cbbc"
Jul 15 21:29:42.268220 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:29:42.268199    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f0d72310abe0e9e85de2f5634edd56fc7d1f01cd197ff135d0426f54df9cbbc\": container with ID starting with 3f0d72310abe0e9e85de2f5634edd56fc7d1f01cd197ff135d0426f54df9cbbc not found: ID does not exist" containerID="3f0d72310abe0e9e85de2f5634edd56fc7d1f01cd197ff135d0426f54df9cbbc"
Jul 15 21:29:42.268285 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:42.268228    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f0d72310abe0e9e85de2f5634edd56fc7d1f01cd197ff135d0426f54df9cbbc"} err="failed to get container status \"3f0d72310abe0e9e85de2f5634edd56fc7d1f01cd197ff135d0426f54df9cbbc\": rpc error: code = NotFound desc = could not find container \"3f0d72310abe0e9e85de2f5634edd56fc7d1f01cd197ff135d0426f54df9cbbc\": container with ID starting with 3f0d72310abe0e9e85de2f5634edd56fc7d1f01cd197ff135d0426f54df9cbbc not found: ID does not exist"
Jul 15 21:29:42.282977 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:42.282953    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f5fcf1-766d7779b4-7fnfr"]
Jul 15 21:29:42.286666 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:42.286647    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f5fcf1-766d7779b4-7fnfr"]
Jul 15 21:29:42.336686 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:42.336658    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5twh\" (UniqueName: \"kubernetes.io/projected/57ddb433-beb7-4734-a0ba-24fd8106553e-kube-api-access-m5twh\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:29:42.336686 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:42.336682    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57ddb433-beb7-4734-a0ba-24fd8106553e-maas-api-tls\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:29:42.336845 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:42.336693    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57ddb433-beb7-4734-a0ba-24fd8106553e-openshift-service-ca\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:29:43.756098 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:43.756066    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57ddb433-beb7-4734-a0ba-24fd8106553e" path="/var/lib/kubelet/pods/57ddb433-beb7-4734-a0ba-24fd8106553e/volumes"
Jul 15 21:29:46.076973 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:46.076939    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:29:52.389779 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:52.389714    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:29:53.730351 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:53.730315    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:29:56.802204 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:56.802166    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k"]
Jul 15 21:29:56.802677 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:56.802410    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k" podUID="312dafc1-a9a9-44ae-9288-de82c7fb439f" containerName="istio-proxy" containerID="cri-o://24c19f4123d68898adf282bceeb58eb9111d2415624589a277ff227d70517b26" gracePeriod=30
Jul 15 21:29:56.834768 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:56.834726    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:29:57.079285 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:57.079188    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:29:57.899657 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:57.899614    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:29:58.002145 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:58.002112    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:29:59.420165 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:59.420128    2566 patch_prober.go:28] interesting pod/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 15 21:29:59.420576 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:29:59.420194    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k" podUID="312dafc1-a9a9-44ae-9288-de82c7fb439f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 15 21:30:00.147415 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:00.147380    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-843ced-29735850-whcgz"]
Jul 15 21:30:00.147752 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:00.147740    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57ddb433-beb7-4734-a0ba-24fd8106553e" containerName="maas-api"
Jul 15 21:30:00.147801 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:00.147754    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="57ddb433-beb7-4734-a0ba-24fd8106553e" containerName="maas-api"
Jul 15 21:30:00.147841 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:00.147821    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="57ddb433-beb7-4734-a0ba-24fd8106553e" containerName="maas-api"
Jul 15 21:30:00.150938 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:00.150920    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-843ced-29735850-whcgz"
Jul 15 21:30:00.154051 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:00.154026    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zsrf5\""
Jul 15 21:30:00.161612 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:00.161585    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-843ced-29735850-whcgz"]
Jul 15 21:30:00.161744 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:00.161616    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-843ced-29735850-whcgz"
Jul 15 21:30:00.502198 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:00.502174    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-843ced-29735850-whcgz"]
Jul 15 21:30:00.504148 ip-10-0-137-234 kubenswrapper[2566]: W0715 21:30:00.504119    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod569b6a68_a396_4f2d_8510_b67dedfe38dd.slice/crio-062923c84bad1f8b5147cc0aa317a72fa9da574b37b31b5cb5203dae583ac831 WatchSource:0}: Error finding container 062923c84bad1f8b5147cc0aa317a72fa9da574b37b31b5cb5203dae583ac831: Status 404 returned error can't find the container with id 062923c84bad1f8b5147cc0aa317a72fa9da574b37b31b5cb5203dae583ac831
Jul 15 21:30:01.354576 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:01.354541    2566 generic.go:356] "Generic (PLEG): container finished" podID="569b6a68-a396-4f2d-8510-b67dedfe38dd" containerID="58526e7b647bc5cdb2faf3e2c709054147ba2c5446f8a03fba0c6e7221a9ec18" exitCode=0
Jul 15 21:30:01.354717 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:01.354605    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-843ced-29735850-whcgz" event={"ID":"569b6a68-a396-4f2d-8510-b67dedfe38dd","Type":"ContainerDied","Data":"58526e7b647bc5cdb2faf3e2c709054147ba2c5446f8a03fba0c6e7221a9ec18"}
Jul 15 21:30:01.354717 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:01.354654    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-843ced-29735850-whcgz" event={"ID":"569b6a68-a396-4f2d-8510-b67dedfe38dd","Type":"ContainerStarted","Data":"062923c84bad1f8b5147cc0aa317a72fa9da574b37b31b5cb5203dae583ac831"}
Jul 15 21:30:02.050438 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:02.050409    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k"
Jul 15 21:30:02.097641 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:02.097558    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/312dafc1-a9a9-44ae-9288-de82c7fb439f-istiod-ca-cert\") pod \"312dafc1-a9a9-44ae-9288-de82c7fb439f\" (UID: \"312dafc1-a9a9-44ae-9288-de82c7fb439f\") "
Jul 15 21:30:02.097641 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:02.097596    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/312dafc1-a9a9-44ae-9288-de82c7fb439f-workload-certs\") pod \"312dafc1-a9a9-44ae-9288-de82c7fb439f\" (UID: \"312dafc1-a9a9-44ae-9288-de82c7fb439f\") "
Jul 15 21:30:02.097641 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:02.097623    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/312dafc1-a9a9-44ae-9288-de82c7fb439f-credential-socket\") pod \"312dafc1-a9a9-44ae-9288-de82c7fb439f\" (UID: \"312dafc1-a9a9-44ae-9288-de82c7fb439f\") "
Jul 15 21:30:02.097933 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:02.097660    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/312dafc1-a9a9-44ae-9288-de82c7fb439f-istio-envoy\") pod \"312dafc1-a9a9-44ae-9288-de82c7fb439f\" (UID: \"312dafc1-a9a9-44ae-9288-de82c7fb439f\") "
Jul 15 21:30:02.097933 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:02.097682    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/312dafc1-a9a9-44ae-9288-de82c7fb439f-istio-podinfo\") pod \"312dafc1-a9a9-44ae-9288-de82c7fb439f\" (UID: \"312dafc1-a9a9-44ae-9288-de82c7fb439f\") "
Jul 15 21:30:02.097933 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:02.097704    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hdlm\" (UniqueName: \"kubernetes.io/projected/312dafc1-a9a9-44ae-9288-de82c7fb439f-kube-api-access-2hdlm\") pod \"312dafc1-a9a9-44ae-9288-de82c7fb439f\" (UID: \"312dafc1-a9a9-44ae-9288-de82c7fb439f\") "
Jul 15 21:30:02.097933 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:02.097781    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/312dafc1-a9a9-44ae-9288-de82c7fb439f-istio-data\") pod \"312dafc1-a9a9-44ae-9288-de82c7fb439f\" (UID: \"312dafc1-a9a9-44ae-9288-de82c7fb439f\") "
Jul 15 21:30:02.097933 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:02.097804    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/312dafc1-a9a9-44ae-9288-de82c7fb439f-workload-socket\") pod \"312dafc1-a9a9-44ae-9288-de82c7fb439f\" (UID: \"312dafc1-a9a9-44ae-9288-de82c7fb439f\") "
Jul 15 21:30:02.097933 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:02.097859    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/312dafc1-a9a9-44ae-9288-de82c7fb439f-istio-token\") pod \"312dafc1-a9a9-44ae-9288-de82c7fb439f\" (UID: \"312dafc1-a9a9-44ae-9288-de82c7fb439f\") "
Jul 15 21:30:02.098236 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:02.098081    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/312dafc1-a9a9-44ae-9288-de82c7fb439f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "312dafc1-a9a9-44ae-9288-de82c7fb439f" (UID: "312dafc1-a9a9-44ae-9288-de82c7fb439f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 21:30:02.098236 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:02.098207    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/312dafc1-a9a9-44ae-9288-de82c7fb439f-istiod-ca-cert\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:30:02.098414 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:02.098325    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/312dafc1-a9a9-44ae-9288-de82c7fb439f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "312dafc1-a9a9-44ae-9288-de82c7fb439f" (UID: "312dafc1-a9a9-44ae-9288-de82c7fb439f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:30:02.098414 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:02.098259    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/312dafc1-a9a9-44ae-9288-de82c7fb439f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "312dafc1-a9a9-44ae-9288-de82c7fb439f" (UID: "312dafc1-a9a9-44ae-9288-de82c7fb439f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:30:02.098532 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:02.098504    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/312dafc1-a9a9-44ae-9288-de82c7fb439f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "312dafc1-a9a9-44ae-9288-de82c7fb439f" (UID: "312dafc1-a9a9-44ae-9288-de82c7fb439f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:30:02.099112 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:02.099086    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/312dafc1-a9a9-44ae-9288-de82c7fb439f-istio-data" (OuterVolumeSpecName: "istio-data") pod "312dafc1-a9a9-44ae-9288-de82c7fb439f" (UID: "312dafc1-a9a9-44ae-9288-de82c7fb439f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:30:02.100364 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:02.100338    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/312dafc1-a9a9-44ae-9288-de82c7fb439f-istio-token" (OuterVolumeSpecName: "istio-token") pod "312dafc1-a9a9-44ae-9288-de82c7fb439f" (UID: "312dafc1-a9a9-44ae-9288-de82c7fb439f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 21:30:02.100364 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:02.100336    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/312dafc1-a9a9-44ae-9288-de82c7fb439f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "312dafc1-a9a9-44ae-9288-de82c7fb439f" (UID: "312dafc1-a9a9-44ae-9288-de82c7fb439f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 15 21:30:02.100546 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:02.100426    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/312dafc1-a9a9-44ae-9288-de82c7fb439f-kube-api-access-2hdlm" (OuterVolumeSpecName: "kube-api-access-2hdlm") pod "312dafc1-a9a9-44ae-9288-de82c7fb439f" (UID: "312dafc1-a9a9-44ae-9288-de82c7fb439f"). InnerVolumeSpecName "kube-api-access-2hdlm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 21:30:02.100605 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:02.100555    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/312dafc1-a9a9-44ae-9288-de82c7fb439f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "312dafc1-a9a9-44ae-9288-de82c7fb439f" (UID: "312dafc1-a9a9-44ae-9288-de82c7fb439f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:30:02.198656 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:02.198625    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/312dafc1-a9a9-44ae-9288-de82c7fb439f-workload-socket\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:30:02.198656 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:02.198653    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/312dafc1-a9a9-44ae-9288-de82c7fb439f-istio-token\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:30:02.198656 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:02.198663    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/312dafc1-a9a9-44ae-9288-de82c7fb439f-workload-certs\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:30:02.198885 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:02.198673    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/312dafc1-a9a9-44ae-9288-de82c7fb439f-credential-socket\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:30:02.198885 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:02.198681    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/312dafc1-a9a9-44ae-9288-de82c7fb439f-istio-envoy\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:30:02.198885 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:02.198689    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/312dafc1-a9a9-44ae-9288-de82c7fb439f-istio-podinfo\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:30:02.198885 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:02.198697    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hdlm\" (UniqueName: \"kubernetes.io/projected/312dafc1-a9a9-44ae-9288-de82c7fb439f-kube-api-access-2hdlm\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:30:02.198885 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:02.198706    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/312dafc1-a9a9-44ae-9288-de82c7fb439f-istio-data\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:30:02.361451 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:02.361355    2566 generic.go:356] "Generic (PLEG): container finished" podID="312dafc1-a9a9-44ae-9288-de82c7fb439f" containerID="24c19f4123d68898adf282bceeb58eb9111d2415624589a277ff227d70517b26" exitCode=0
Jul 15 21:30:02.361451 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:02.361406    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k"
Jul 15 21:30:02.361451 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:02.361440    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k" event={"ID":"312dafc1-a9a9-44ae-9288-de82c7fb439f","Type":"ContainerDied","Data":"24c19f4123d68898adf282bceeb58eb9111d2415624589a277ff227d70517b26"}
Jul 15 21:30:02.361707 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:02.361495    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k" event={"ID":"312dafc1-a9a9-44ae-9288-de82c7fb439f","Type":"ContainerDied","Data":"1ba84374f6c990ded86acdd9461a3ce1c36f47d07a5f542363651c777b4434ee"}
Jul 15 21:30:02.361707 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:02.361519    2566 scope.go:117] "RemoveContainer" containerID="24c19f4123d68898adf282bceeb58eb9111d2415624589a277ff227d70517b26"
Jul 15 21:30:02.372290 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:02.372270    2566 scope.go:117] "RemoveContainer" containerID="514b3d940dd98131d2e7fcb687d6965a90fe3381774ebf23624fdc9ebf88336b"
Jul 15 21:30:02.380972 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:02.380953    2566 scope.go:117] "RemoveContainer" containerID="24c19f4123d68898adf282bceeb58eb9111d2415624589a277ff227d70517b26"
Jul 15 21:30:02.381312 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:30:02.381283    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24c19f4123d68898adf282bceeb58eb9111d2415624589a277ff227d70517b26\": container with ID starting with 24c19f4123d68898adf282bceeb58eb9111d2415624589a277ff227d70517b26 not found: ID does not exist" containerID="24c19f4123d68898adf282bceeb58eb9111d2415624589a277ff227d70517b26"
Jul 15 21:30:02.381376 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:02.381313    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24c19f4123d68898adf282bceeb58eb9111d2415624589a277ff227d70517b26"} err="failed to get container status \"24c19f4123d68898adf282bceeb58eb9111d2415624589a277ff227d70517b26\": rpc error: code = NotFound desc = could not find container \"24c19f4123d68898adf282bceeb58eb9111d2415624589a277ff227d70517b26\": container with ID starting with 24c19f4123d68898adf282bceeb58eb9111d2415624589a277ff227d70517b26 not found: ID does not exist"
Jul 15 21:30:02.381376 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:02.381333    2566 scope.go:117] "RemoveContainer" containerID="514b3d940dd98131d2e7fcb687d6965a90fe3381774ebf23624fdc9ebf88336b"
Jul 15 21:30:02.381586 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:30:02.381568    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"514b3d940dd98131d2e7fcb687d6965a90fe3381774ebf23624fdc9ebf88336b\": container with ID starting with 514b3d940dd98131d2e7fcb687d6965a90fe3381774ebf23624fdc9ebf88336b not found: ID does not exist" containerID="514b3d940dd98131d2e7fcb687d6965a90fe3381774ebf23624fdc9ebf88336b"
Jul 15 21:30:02.381636 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:02.381589    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"514b3d940dd98131d2e7fcb687d6965a90fe3381774ebf23624fdc9ebf88336b"} err="failed to get container status \"514b3d940dd98131d2e7fcb687d6965a90fe3381774ebf23624fdc9ebf88336b\": rpc error: code = NotFound desc = could not find container \"514b3d940dd98131d2e7fcb687d6965a90fe3381774ebf23624fdc9ebf88336b\": container with ID starting with 514b3d940dd98131d2e7fcb687d6965a90fe3381774ebf23624fdc9ebf88336b not found: ID does not exist"
Jul 15 21:30:02.385405 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:02.385381    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k"]
Jul 15 21:30:02.391333 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:02.391310    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f5fcf1-openshift-default-5dc54b57d5-9zt9k"]
Jul 15 21:30:02.503216 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:02.503192    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-843ced-29735850-whcgz"
Jul 15 21:30:03.367674 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:03.367644    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-843ced-29735850-whcgz"
Jul 15 21:30:03.367674 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:03.367667    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-843ced-29735850-whcgz" event={"ID":"569b6a68-a396-4f2d-8510-b67dedfe38dd","Type":"ContainerDied","Data":"062923c84bad1f8b5147cc0aa317a72fa9da574b37b31b5cb5203dae583ac831"}
Jul 15 21:30:03.368280 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:03.367715    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="062923c84bad1f8b5147cc0aa317a72fa9da574b37b31b5cb5203dae583ac831"
Jul 15 21:30:03.757053 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:03.757014    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="312dafc1-a9a9-44ae-9288-de82c7fb439f" path="/var/lib/kubelet/pods/312dafc1-a9a9-44ae-9288-de82c7fb439f/volumes"
Jul 15 21:30:07.726802 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:07.726763    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-843ced-fb5f4b8cf-zqpsf"]
Jul 15 21:30:07.727619 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:07.727580    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-843ced-fb5f4b8cf-zqpsf" podUID="4dae6215-3e23-49a1-879d-26881faae017" containerName="maas-api" containerID="cri-o://d30ab4b793e85148af19b501e1d60e866b2d842bb207be6d81bf748660f04ee0" gracePeriod=30
Jul 15 21:30:07.764935 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:07.764907    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:30:07.765342 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:07.765299    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-843ced-29735850-whcgz"]
Jul 15 21:30:07.769348 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:07.769328    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-843ced-29735850-whcgz"]
Jul 15 21:30:08.005477 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:08.005443    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-843ced-fb5f4b8cf-zqpsf"
Jul 15 21:30:08.052606 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:08.052576    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nftd6\" (UniqueName: \"kubernetes.io/projected/4dae6215-3e23-49a1-879d-26881faae017-kube-api-access-nftd6\") pod \"4dae6215-3e23-49a1-879d-26881faae017\" (UID: \"4dae6215-3e23-49a1-879d-26881faae017\") "
Jul 15 21:30:08.052787 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:08.052633    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dae6215-3e23-49a1-879d-26881faae017-openshift-service-ca\") pod \"4dae6215-3e23-49a1-879d-26881faae017\" (UID: \"4dae6215-3e23-49a1-879d-26881faae017\") "
Jul 15 21:30:08.052787 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:08.052728    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dae6215-3e23-49a1-879d-26881faae017-maas-api-tls\") pod \"4dae6215-3e23-49a1-879d-26881faae017\" (UID: \"4dae6215-3e23-49a1-879d-26881faae017\") "
Jul 15 21:30:08.053056 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:08.053032    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dae6215-3e23-49a1-879d-26881faae017-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4dae6215-3e23-49a1-879d-26881faae017" (UID: "4dae6215-3e23-49a1-879d-26881faae017"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 21:30:08.054743 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:08.054718    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dae6215-3e23-49a1-879d-26881faae017-kube-api-access-nftd6" (OuterVolumeSpecName: "kube-api-access-nftd6") pod "4dae6215-3e23-49a1-879d-26881faae017" (UID: "4dae6215-3e23-49a1-879d-26881faae017"). InnerVolumeSpecName "kube-api-access-nftd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 21:30:08.054919 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:08.054868    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dae6215-3e23-49a1-879d-26881faae017-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4dae6215-3e23-49a1-879d-26881faae017" (UID: "4dae6215-3e23-49a1-879d-26881faae017"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 21:30:08.153409 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:08.153379    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dae6215-3e23-49a1-879d-26881faae017-maas-api-tls\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:30:08.153409 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:08.153406    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nftd6\" (UniqueName: \"kubernetes.io/projected/4dae6215-3e23-49a1-879d-26881faae017-kube-api-access-nftd6\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:30:08.153409 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:08.153415    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dae6215-3e23-49a1-879d-26881faae017-openshift-service-ca\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:30:08.394232 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:08.394143    2566 generic.go:356] "Generic (PLEG): container finished" podID="4dae6215-3e23-49a1-879d-26881faae017" containerID="d30ab4b793e85148af19b501e1d60e866b2d842bb207be6d81bf748660f04ee0" exitCode=0
Jul 15 21:30:08.394232 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:08.394192    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-843ced-fb5f4b8cf-zqpsf"
Jul 15 21:30:08.394232 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:08.394222    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-843ced-fb5f4b8cf-zqpsf" event={"ID":"4dae6215-3e23-49a1-879d-26881faae017","Type":"ContainerDied","Data":"d30ab4b793e85148af19b501e1d60e866b2d842bb207be6d81bf748660f04ee0"}
Jul 15 21:30:08.394481 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:08.394278    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-843ced-fb5f4b8cf-zqpsf" event={"ID":"4dae6215-3e23-49a1-879d-26881faae017","Type":"ContainerDied","Data":"1a6b198e29dc4da096fd4f5af328a16bcec6037d5913354c7c378305a692365d"}
Jul 15 21:30:08.394481 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:08.394299    2566 scope.go:117] "RemoveContainer" containerID="d30ab4b793e85148af19b501e1d60e866b2d842bb207be6d81bf748660f04ee0"
Jul 15 21:30:08.403321 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:08.403304    2566 scope.go:117] "RemoveContainer" containerID="d30ab4b793e85148af19b501e1d60e866b2d842bb207be6d81bf748660f04ee0"
Jul 15 21:30:08.403582 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:30:08.403564    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d30ab4b793e85148af19b501e1d60e866b2d842bb207be6d81bf748660f04ee0\": container with ID starting with d30ab4b793e85148af19b501e1d60e866b2d842bb207be6d81bf748660f04ee0 not found: ID does not exist" containerID="d30ab4b793e85148af19b501e1d60e866b2d842bb207be6d81bf748660f04ee0"
Jul 15 21:30:08.403659 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:08.403588    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d30ab4b793e85148af19b501e1d60e866b2d842bb207be6d81bf748660f04ee0"} err="failed to get container status \"d30ab4b793e85148af19b501e1d60e866b2d842bb207be6d81bf748660f04ee0\": rpc error: code = NotFound desc = could not find container \"d30ab4b793e85148af19b501e1d60e866b2d842bb207be6d81bf748660f04ee0\": container with ID starting with d30ab4b793e85148af19b501e1d60e866b2d842bb207be6d81bf748660f04ee0 not found: ID does not exist"
Jul 15 21:30:08.417790 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:08.417764    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-843ced-fb5f4b8cf-zqpsf"]
Jul 15 21:30:08.423787 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:08.423767    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-843ced-fb5f4b8cf-zqpsf"]
Jul 15 21:30:09.756741 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:09.756706    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4dae6215-3e23-49a1-879d-26881faae017" path="/var/lib/kubelet/pods/4dae6215-3e23-49a1-879d-26881faae017/volumes"
Jul 15 21:30:09.757234 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:09.757205    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="569b6a68-a396-4f2d-8510-b67dedfe38dd" path="/var/lib/kubelet/pods/569b6a68-a396-4f2d-8510-b67dedfe38dd/volumes"
Jul 15 21:30:12.319271 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:12.319233    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:30:18.604440 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:18.604403    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:30:18.704882 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:18.704845    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:30:23.045776 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:23.045741    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh"]
Jul 15 21:30:23.046222 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:23.045960    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh" podUID="570f9db7-c9cd-42f6-b044-ce7069a322ae" containerName="istio-proxy" containerID="cri-o://a20c154e0d5e32c9d4df5a908a02052d6f20f76a90ef6f76e548f5afe729a404" gracePeriod=30
Jul 15 21:30:23.066718 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:23.066693    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:30:23.322456 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:23.322366    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:30:23.726828 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:23.726795    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:30:24.037709 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:24.037623    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp"]
Jul 15 21:30:24.038098 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:24.038078    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="312dafc1-a9a9-44ae-9288-de82c7fb439f" containerName="istio-proxy"
Jul 15 21:30:24.038098 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:24.038099    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="312dafc1-a9a9-44ae-9288-de82c7fb439f" containerName="istio-proxy"
Jul 15 21:30:24.038254 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:24.038115    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="312dafc1-a9a9-44ae-9288-de82c7fb439f" containerName="istio-proxy"
Jul 15 21:30:24.038254 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:24.038123    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="312dafc1-a9a9-44ae-9288-de82c7fb439f" containerName="istio-proxy"
Jul 15 21:30:24.038254 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:24.038141    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dae6215-3e23-49a1-879d-26881faae017" containerName="maas-api"
Jul 15 21:30:24.038254 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:24.038150    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dae6215-3e23-49a1-879d-26881faae017" containerName="maas-api"
Jul 15 21:30:24.038254 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:24.038159    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="569b6a68-a396-4f2d-8510-b67dedfe38dd" containerName="cleanup"
Jul 15 21:30:24.038254 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:24.038167    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="569b6a68-a396-4f2d-8510-b67dedfe38dd" containerName="cleanup"
Jul 15 21:30:24.038551 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:24.038275    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dae6215-3e23-49a1-879d-26881faae017" containerName="maas-api"
Jul 15 21:30:24.038551 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:24.038288    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="312dafc1-a9a9-44ae-9288-de82c7fb439f" containerName="istio-proxy"
Jul 15 21:30:24.038551 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:24.038298    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="312dafc1-a9a9-44ae-9288-de82c7fb439f" containerName="istio-proxy"
Jul 15 21:30:24.038551 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:24.038308    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="569b6a68-a396-4f2d-8510-b67dedfe38dd" containerName="cleanup"
Jul 15 21:30:24.041490 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:24.041466    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp"
Jul 15 21:30:24.050122 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:24.050100    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-677131-openshift-default-dockercfg-wc6t6\""
Jul 15 21:30:24.075961 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:24.075935    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp"]
Jul 15 21:30:24.095536 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:24.095508    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:30:24.193126 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:24.193094    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a48bab9d-061c-45b7-8b8a-8e343d64058a-istiod-ca-cert\") pod \"e2e-shared-a-677131-openshift-default-784d56455-tgbzp\" (UID: \"a48bab9d-061c-45b7-8b8a-8e343d64058a\") " pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp"
Jul 15 21:30:24.193300 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:24.193132    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a48bab9d-061c-45b7-8b8a-8e343d64058a-istio-data\") pod \"e2e-shared-a-677131-openshift-default-784d56455-tgbzp\" (UID: \"a48bab9d-061c-45b7-8b8a-8e343d64058a\") " pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp"
Jul 15 21:30:24.193300 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:24.193238    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a48bab9d-061c-45b7-8b8a-8e343d64058a-credential-socket\") pod \"e2e-shared-a-677131-openshift-default-784d56455-tgbzp\" (UID: \"a48bab9d-061c-45b7-8b8a-8e343d64058a\") " pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp"
Jul 15 21:30:24.193300 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:24.193266    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a48bab9d-061c-45b7-8b8a-8e343d64058a-istio-podinfo\") pod \"e2e-shared-a-677131-openshift-default-784d56455-tgbzp\" (UID: \"a48bab9d-061c-45b7-8b8a-8e343d64058a\") " pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp"
Jul 15 21:30:24.193466 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:24.193305    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a48bab9d-061c-45b7-8b8a-8e343d64058a-istio-token\") pod \"e2e-shared-a-677131-openshift-default-784d56455-tgbzp\" (UID: \"a48bab9d-061c-45b7-8b8a-8e343d64058a\") " pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp"
Jul 15 21:30:24.193466 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:24.193405    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a48bab9d-061c-45b7-8b8a-8e343d64058a-workload-certs\") pod \"e2e-shared-a-677131-openshift-default-784d56455-tgbzp\" (UID: \"a48bab9d-061c-45b7-8b8a-8e343d64058a\") " pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp"
Jul 15 21:30:24.193570 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:24.193466    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a48bab9d-061c-45b7-8b8a-8e343d64058a-istio-envoy\") pod \"e2e-shared-a-677131-openshift-default-784d56455-tgbzp\" (UID: \"a48bab9d-061c-45b7-8b8a-8e343d64058a\") " pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp"
Jul 15 21:30:24.193570 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:24.193493    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2jgg\" (UniqueName: \"kubernetes.io/projected/a48bab9d-061c-45b7-8b8a-8e343d64058a-kube-api-access-m2jgg\") pod \"e2e-shared-a-677131-openshift-default-784d56455-tgbzp\" (UID: \"a48bab9d-061c-45b7-8b8a-8e343d64058a\") " pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp"
Jul 15 21:30:24.193661 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:24.193573    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a48bab9d-061c-45b7-8b8a-8e343d64058a-workload-socket\") pod \"e2e-shared-a-677131-openshift-default-784d56455-tgbzp\" (UID: \"a48bab9d-061c-45b7-8b8a-8e343d64058a\") " pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp"
Jul 15 21:30:24.294099 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:24.294004    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a48bab9d-061c-45b7-8b8a-8e343d64058a-istiod-ca-cert\") pod \"e2e-shared-a-677131-openshift-default-784d56455-tgbzp\" (UID: \"a48bab9d-061c-45b7-8b8a-8e343d64058a\") " pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp"
Jul 15 21:30:24.294099 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:24.294050    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a48bab9d-061c-45b7-8b8a-8e343d64058a-istio-data\") pod \"e2e-shared-a-677131-openshift-default-784d56455-tgbzp\" (UID: \"a48bab9d-061c-45b7-8b8a-8e343d64058a\") " pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp"
Jul 15 21:30:24.294340 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:24.294102    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a48bab9d-061c-45b7-8b8a-8e343d64058a-credential-socket\") pod \"e2e-shared-a-677131-openshift-default-784d56455-tgbzp\" (UID: \"a48bab9d-061c-45b7-8b8a-8e343d64058a\") " pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp"
Jul 15 21:30:24.294340 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:24.294124    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a48bab9d-061c-45b7-8b8a-8e343d64058a-istio-podinfo\") pod \"e2e-shared-a-677131-openshift-default-784d56455-tgbzp\" (UID: \"a48bab9d-061c-45b7-8b8a-8e343d64058a\") " pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp"
Jul 15 21:30:24.294340 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:24.294285    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a48bab9d-061c-45b7-8b8a-8e343d64058a-istio-token\") pod \"e2e-shared-a-677131-openshift-default-784d56455-tgbzp\" (UID: \"a48bab9d-061c-45b7-8b8a-8e343d64058a\") " pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp"
Jul 15 21:30:24.294512 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:24.294344    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a48bab9d-061c-45b7-8b8a-8e343d64058a-workload-certs\") pod \"e2e-shared-a-677131-openshift-default-784d56455-tgbzp\" (UID: \"a48bab9d-061c-45b7-8b8a-8e343d64058a\") " pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp"
Jul 15 21:30:24.294512 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:24.294398    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a48bab9d-061c-45b7-8b8a-8e343d64058a-istio-envoy\") pod \"e2e-shared-a-677131-openshift-default-784d56455-tgbzp\" (UID: \"a48bab9d-061c-45b7-8b8a-8e343d64058a\") " pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp"
Jul 15 21:30:24.294512 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:24.294425    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2jgg\" (UniqueName: \"kubernetes.io/projected/a48bab9d-061c-45b7-8b8a-8e343d64058a-kube-api-access-m2jgg\") pod \"e2e-shared-a-677131-openshift-default-784d56455-tgbzp\" (UID: \"a48bab9d-061c-45b7-8b8a-8e343d64058a\") " pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp"
Jul 15 21:30:24.294512 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:24.294495    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a48bab9d-061c-45b7-8b8a-8e343d64058a-workload-socket\") pod \"e2e-shared-a-677131-openshift-default-784d56455-tgbzp\" (UID: \"a48bab9d-061c-45b7-8b8a-8e343d64058a\") " pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp"
Jul 15 21:30:24.294719 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:24.294525    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a48bab9d-061c-45b7-8b8a-8e343d64058a-credential-socket\") pod \"e2e-shared-a-677131-openshift-default-784d56455-tgbzp\" (UID: \"a48bab9d-061c-45b7-8b8a-8e343d64058a\") " pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp"
Jul 15 21:30:24.294719 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:24.294584    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a48bab9d-061c-45b7-8b8a-8e343d64058a-istio-data\") pod \"e2e-shared-a-677131-openshift-default-784d56455-tgbzp\" (UID: \"a48bab9d-061c-45b7-8b8a-8e343d64058a\") " pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp"
Jul 15 21:30:24.294851 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:24.294824    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a48bab9d-061c-45b7-8b8a-8e343d64058a-istiod-ca-cert\") pod \"e2e-shared-a-677131-openshift-default-784d56455-tgbzp\" (UID: \"a48bab9d-061c-45b7-8b8a-8e343d64058a\") " pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp"
Jul 15 21:30:24.295050 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:24.294869    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a48bab9d-061c-45b7-8b8a-8e343d64058a-workload-socket\") pod \"e2e-shared-a-677131-openshift-default-784d56455-tgbzp\" (UID: \"a48bab9d-061c-45b7-8b8a-8e343d64058a\") " pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp"
Jul 15 21:30:24.295126 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:24.294975    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a48bab9d-061c-45b7-8b8a-8e343d64058a-workload-certs\") pod \"e2e-shared-a-677131-openshift-default-784d56455-tgbzp\" (UID: \"a48bab9d-061c-45b7-8b8a-8e343d64058a\") " pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp"
Jul 15 21:30:24.296821 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:24.296786    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a48bab9d-061c-45b7-8b8a-8e343d64058a-istio-podinfo\") pod \"e2e-shared-a-677131-openshift-default-784d56455-tgbzp\" (UID: \"a48bab9d-061c-45b7-8b8a-8e343d64058a\") " pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp"
Jul 15 21:30:24.297003 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:24.296984    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a48bab9d-061c-45b7-8b8a-8e343d64058a-istio-envoy\") pod \"e2e-shared-a-677131-openshift-default-784d56455-tgbzp\" (UID: \"a48bab9d-061c-45b7-8b8a-8e343d64058a\") " pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp"
Jul 15 21:30:24.304249 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:24.304226    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a48bab9d-061c-45b7-8b8a-8e343d64058a-istio-token\") pod \"e2e-shared-a-677131-openshift-default-784d56455-tgbzp\" (UID: \"a48bab9d-061c-45b7-8b8a-8e343d64058a\") " pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp"
Jul 15 21:30:24.304356 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:24.304300    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2jgg\" (UniqueName: \"kubernetes.io/projected/a48bab9d-061c-45b7-8b8a-8e343d64058a-kube-api-access-m2jgg\") pod \"e2e-shared-a-677131-openshift-default-784d56455-tgbzp\" (UID: \"a48bab9d-061c-45b7-8b8a-8e343d64058a\") " pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp"
Jul 15 21:30:24.351940 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:24.351887    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp"
Jul 15 21:30:24.412419 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:24.412354    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:30:24.494554 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:24.494528    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp"]
Jul 15 21:30:24.496128 ip-10-0-137-234 kubenswrapper[2566]: W0715 21:30:24.496092    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda48bab9d_061c_45b7_8b8a_8e343d64058a.slice/crio-3e73a9eaa12c26a06767142a515734dac50f09e2f2fef77b699fc2030ea8611b WatchSource:0}: Error finding container 3e73a9eaa12c26a06767142a515734dac50f09e2f2fef77b699fc2030ea8611b: Status 404 returned error can't find the container with id 3e73a9eaa12c26a06767142a515734dac50f09e2f2fef77b699fc2030ea8611b
Jul 15 21:30:24.498060 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:24.498025    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:30:24.498175 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:24.498093    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:30:24.498175 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:24.498120    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:30:25.485424 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:25.485387    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp" event={"ID":"a48bab9d-061c-45b7-8b8a-8e343d64058a","Type":"ContainerStarted","Data":"a8bdbdde8a0aeabfa114107ece61bcd6be780de2889913a700da6d69c09de98a"}
Jul 15 21:30:25.485887 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:25.485447    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp" event={"ID":"a48bab9d-061c-45b7-8b8a-8e343d64058a","Type":"ContainerStarted","Data":"3e73a9eaa12c26a06767142a515734dac50f09e2f2fef77b699fc2030ea8611b"}
Jul 15 21:30:25.514321 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:25.514263    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp" podStartSLOduration=1.514249157 podStartE2EDuration="1.514249157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:30:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:30:25.512539631 +0000 UTC m=+2558.661577689" watchObservedRunningTime="2026-07-15 21:30:25.514249157 +0000 UTC m=+2558.663287213"
Jul 15 21:30:26.352859 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:26.352826    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp"
Jul 15 21:30:26.357964 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:26.357936    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp"
Jul 15 21:30:26.490406 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:26.490371    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp"
Jul 15 21:30:26.491433 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:26.491414    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp"
Jul 15 21:30:27.520742 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:27.520702    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:30:27.624187 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:27.624149    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:30:27.791566 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:27.791473    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-677131-597975fd87-h7pjv"]
Jul 15 21:30:27.796766 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:27.796736    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-677131-597975fd87-h7pjv"
Jul 15 21:30:27.800550 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:27.800525    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-s2l9d\""
Jul 15 21:30:27.800804 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:27.800785    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-677131-serving-cert\""
Jul 15 21:30:27.801838 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:27.801815    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 15 21:30:27.801960 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:27.801861    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 15 21:30:27.823465 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:27.823439    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-677131-597975fd87-h7pjv"]
Jul 15 21:30:27.837856 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:27.837823    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:30:27.924472 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:27.924434    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq888\" (UniqueName: \"kubernetes.io/projected/8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f-kube-api-access-bq888\") pod \"maas-api-e2e-shared-a-677131-597975fd87-h7pjv\" (UID: \"8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-677131-597975fd87-h7pjv"
Jul 15 21:30:27.924647 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:27.924569    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f-maas-api-tls\") pod \"maas-api-e2e-shared-a-677131-597975fd87-h7pjv\" (UID: \"8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-677131-597975fd87-h7pjv"
Jul 15 21:30:27.924647 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:27.924638    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-677131-597975fd87-h7pjv\" (UID: \"8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-677131-597975fd87-h7pjv"
Jul 15 21:30:28.026034 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:28.025993    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-677131-597975fd87-h7pjv\" (UID: \"8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-677131-597975fd87-h7pjv"
Jul 15 21:30:28.026225 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:28.026069    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq888\" (UniqueName: \"kubernetes.io/projected/8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f-kube-api-access-bq888\") pod \"maas-api-e2e-shared-a-677131-597975fd87-h7pjv\" (UID: \"8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-677131-597975fd87-h7pjv"
Jul 15 21:30:28.026225 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:28.026143    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f-maas-api-tls\") pod \"maas-api-e2e-shared-a-677131-597975fd87-h7pjv\" (UID: \"8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-677131-597975fd87-h7pjv"
Jul 15 21:30:28.026341 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:30:28.026229    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-677131-serving-cert: secret "maas-api-e2e-shared-a-677131-serving-cert" not found
Jul 15 21:30:28.026341 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:30:28.026304    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f-maas-api-tls podName:8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f nodeName:}" failed. No retries permitted until 2026-07-15 21:30:28.526287925 +0000 UTC m=+2561.675325960 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f-maas-api-tls") pod "maas-api-e2e-shared-a-677131-597975fd87-h7pjv" (UID: "8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f") : secret "maas-api-e2e-shared-a-677131-serving-cert" not found
Jul 15 21:30:28.026811 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:28.026788    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-677131-597975fd87-h7pjv\" (UID: \"8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-677131-597975fd87-h7pjv"
Jul 15 21:30:28.035686 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:28.035664    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq888\" (UniqueName: \"kubernetes.io/projected/8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f-kube-api-access-bq888\") pod \"maas-api-e2e-shared-a-677131-597975fd87-h7pjv\" (UID: \"8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-677131-597975fd87-h7pjv"
Jul 15 21:30:28.277760 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:28.277668    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:30:28.310120 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:28.310061    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh"
Jul 15 21:30:28.408313 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:28.408276    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:30:28.429141 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:28.429115    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/570f9db7-c9cd-42f6-b044-ce7069a322ae-istio-data\") pod \"570f9db7-c9cd-42f6-b044-ce7069a322ae\" (UID: \"570f9db7-c9cd-42f6-b044-ce7069a322ae\") "
Jul 15 21:30:28.429298 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:28.429149    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/570f9db7-c9cd-42f6-b044-ce7069a322ae-istiod-ca-cert\") pod \"570f9db7-c9cd-42f6-b044-ce7069a322ae\" (UID: \"570f9db7-c9cd-42f6-b044-ce7069a322ae\") "
Jul 15 21:30:28.429298 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:28.429190    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6x59\" (UniqueName: \"kubernetes.io/projected/570f9db7-c9cd-42f6-b044-ce7069a322ae-kube-api-access-r6x59\") pod \"570f9db7-c9cd-42f6-b044-ce7069a322ae\" (UID: \"570f9db7-c9cd-42f6-b044-ce7069a322ae\") "
Jul 15 21:30:28.429298 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:28.429211    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/570f9db7-c9cd-42f6-b044-ce7069a322ae-istio-token\") pod \"570f9db7-c9cd-42f6-b044-ce7069a322ae\" (UID: \"570f9db7-c9cd-42f6-b044-ce7069a322ae\") "
Jul 15 21:30:28.429298 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:28.429248    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/570f9db7-c9cd-42f6-b044-ce7069a322ae-workload-socket\") pod \"570f9db7-c9cd-42f6-b044-ce7069a322ae\" (UID: \"570f9db7-c9cd-42f6-b044-ce7069a322ae\") "
Jul 15 21:30:28.429298 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:28.429292    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/570f9db7-c9cd-42f6-b044-ce7069a322ae-istio-podinfo\") pod \"570f9db7-c9cd-42f6-b044-ce7069a322ae\" (UID: \"570f9db7-c9cd-42f6-b044-ce7069a322ae\") "
Jul 15 21:30:28.429563 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:28.429317    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/570f9db7-c9cd-42f6-b044-ce7069a322ae-workload-certs\") pod \"570f9db7-c9cd-42f6-b044-ce7069a322ae\" (UID: \"570f9db7-c9cd-42f6-b044-ce7069a322ae\") "
Jul 15 21:30:28.429563 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:28.429352    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/570f9db7-c9cd-42f6-b044-ce7069a322ae-credential-socket\") pod \"570f9db7-c9cd-42f6-b044-ce7069a322ae\" (UID: \"570f9db7-c9cd-42f6-b044-ce7069a322ae\") "
Jul 15 21:30:28.429563 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:28.429387    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/570f9db7-c9cd-42f6-b044-ce7069a322ae-istio-envoy\") pod \"570f9db7-c9cd-42f6-b044-ce7069a322ae\" (UID: \"570f9db7-c9cd-42f6-b044-ce7069a322ae\") "
Jul 15 21:30:28.429563 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:28.429523    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/570f9db7-c9cd-42f6-b044-ce7069a322ae-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "570f9db7-c9cd-42f6-b044-ce7069a322ae" (UID: "570f9db7-c9cd-42f6-b044-ce7069a322ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:30:28.429821 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:28.429570    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/570f9db7-c9cd-42f6-b044-ce7069a322ae-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "570f9db7-c9cd-42f6-b044-ce7069a322ae" (UID: "570f9db7-c9cd-42f6-b044-ce7069a322ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 21:30:28.429821 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:28.429737    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/570f9db7-c9cd-42f6-b044-ce7069a322ae-workload-socket\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:30:28.429821 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:28.429644    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/570f9db7-c9cd-42f6-b044-ce7069a322ae-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "570f9db7-c9cd-42f6-b044-ce7069a322ae" (UID: "570f9db7-c9cd-42f6-b044-ce7069a322ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:30:28.429821 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:28.429657    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/570f9db7-c9cd-42f6-b044-ce7069a322ae-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "570f9db7-c9cd-42f6-b044-ce7069a322ae" (UID: "570f9db7-c9cd-42f6-b044-ce7069a322ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:30:28.430287 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:28.430263    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/570f9db7-c9cd-42f6-b044-ce7069a322ae-istio-data" (OuterVolumeSpecName: "istio-data") pod "570f9db7-c9cd-42f6-b044-ce7069a322ae" (UID: "570f9db7-c9cd-42f6-b044-ce7069a322ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:30:28.431720 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:28.431698    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/570f9db7-c9cd-42f6-b044-ce7069a322ae-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "570f9db7-c9cd-42f6-b044-ce7069a322ae" (UID: "570f9db7-c9cd-42f6-b044-ce7069a322ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:30:28.431804 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:28.431781    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/570f9db7-c9cd-42f6-b044-ce7069a322ae-kube-api-access-r6x59" (OuterVolumeSpecName: "kube-api-access-r6x59") pod "570f9db7-c9cd-42f6-b044-ce7069a322ae" (UID: "570f9db7-c9cd-42f6-b044-ce7069a322ae"). InnerVolumeSpecName "kube-api-access-r6x59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 21:30:28.432103 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:28.432080    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/570f9db7-c9cd-42f6-b044-ce7069a322ae-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "570f9db7-c9cd-42f6-b044-ce7069a322ae" (UID: "570f9db7-c9cd-42f6-b044-ce7069a322ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 15 21:30:28.432155 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:28.432087    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/570f9db7-c9cd-42f6-b044-ce7069a322ae-istio-token" (OuterVolumeSpecName: "istio-token") pod "570f9db7-c9cd-42f6-b044-ce7069a322ae" (UID: "570f9db7-c9cd-42f6-b044-ce7069a322ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 21:30:28.502498 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:28.502460    2566 generic.go:356] "Generic (PLEG): container finished" podID="570f9db7-c9cd-42f6-b044-ce7069a322ae" containerID="a20c154e0d5e32c9d4df5a908a02052d6f20f76a90ef6f76e548f5afe729a404" exitCode=0
Jul 15 21:30:28.502656 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:28.502515    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh"
Jul 15 21:30:28.502656 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:28.502554    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh" event={"ID":"570f9db7-c9cd-42f6-b044-ce7069a322ae","Type":"ContainerDied","Data":"a20c154e0d5e32c9d4df5a908a02052d6f20f76a90ef6f76e548f5afe729a404"}
Jul 15 21:30:28.502656 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:28.502612    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh" event={"ID":"570f9db7-c9cd-42f6-b044-ce7069a322ae","Type":"ContainerDied","Data":"9772dcc326b72bed5a8e59e0d266d5614d038171a1a743021de0d93fb47e2aef"}
Jul 15 21:30:28.502656 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:28.502644    2566 scope.go:117] "RemoveContainer" containerID="a20c154e0d5e32c9d4df5a908a02052d6f20f76a90ef6f76e548f5afe729a404"
Jul 15 21:30:28.509830 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:28.509797    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:30:28.511611 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:28.511592    2566 scope.go:117] "RemoveContainer" containerID="f6110fecd01c2446d3ca47bd5a56f4a06ec76d014bfee1a74c7e609969295625"
Jul 15 21:30:28.519643 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:28.519625    2566 scope.go:117] "RemoveContainer" containerID="a20c154e0d5e32c9d4df5a908a02052d6f20f76a90ef6f76e548f5afe729a404"
Jul 15 21:30:28.519918 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:30:28.519875    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a20c154e0d5e32c9d4df5a908a02052d6f20f76a90ef6f76e548f5afe729a404\": container with ID starting with a20c154e0d5e32c9d4df5a908a02052d6f20f76a90ef6f76e548f5afe729a404 not found: ID does not exist" containerID="a20c154e0d5e32c9d4df5a908a02052d6f20f76a90ef6f76e548f5afe729a404"
Jul 15 21:30:28.519972 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:28.519923    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a20c154e0d5e32c9d4df5a908a02052d6f20f76a90ef6f76e548f5afe729a404"} err="failed to get container status \"a20c154e0d5e32c9d4df5a908a02052d6f20f76a90ef6f76e548f5afe729a404\": rpc error: code = NotFound desc = could not find container \"a20c154e0d5e32c9d4df5a908a02052d6f20f76a90ef6f76e548f5afe729a404\": container with ID starting with a20c154e0d5e32c9d4df5a908a02052d6f20f76a90ef6f76e548f5afe729a404 not found: ID does not exist"
Jul 15 21:30:28.519972 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:28.519941    2566 scope.go:117] "RemoveContainer" containerID="f6110fecd01c2446d3ca47bd5a56f4a06ec76d014bfee1a74c7e609969295625"
Jul 15 21:30:28.520193 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:30:28.520174    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6110fecd01c2446d3ca47bd5a56f4a06ec76d014bfee1a74c7e609969295625\": container with ID starting with f6110fecd01c2446d3ca47bd5a56f4a06ec76d014bfee1a74c7e609969295625 not found: ID does not exist" containerID="f6110fecd01c2446d3ca47bd5a56f4a06ec76d014bfee1a74c7e609969295625"
Jul 15 21:30:28.520243 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:28.520199    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6110fecd01c2446d3ca47bd5a56f4a06ec76d014bfee1a74c7e609969295625"} err="failed to get container status \"f6110fecd01c2446d3ca47bd5a56f4a06ec76d014bfee1a74c7e609969295625\": rpc error: code = NotFound desc = could not find container \"f6110fecd01c2446d3ca47bd5a56f4a06ec76d014bfee1a74c7e609969295625\": container with ID starting with f6110fecd01c2446d3ca47bd5a56f4a06ec76d014bfee1a74c7e609969295625 not found: ID does not exist"
Jul 15 21:30:28.528234 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:28.528208    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh"]
Jul 15 21:30:28.530361 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:28.530337    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f-maas-api-tls\") pod \"maas-api-e2e-shared-a-677131-597975fd87-h7pjv\" (UID: \"8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-677131-597975fd87-h7pjv"
Jul 15 21:30:28.530506 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:28.530418    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/570f9db7-c9cd-42f6-b044-ce7069a322ae-istio-data\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:30:28.530506 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:28.530434    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/570f9db7-c9cd-42f6-b044-ce7069a322ae-istiod-ca-cert\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:30:28.530643 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:28.530516    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6x59\" (UniqueName: \"kubernetes.io/projected/570f9db7-c9cd-42f6-b044-ce7069a322ae-kube-api-access-r6x59\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:30:28.530643 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:28.530532    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/570f9db7-c9cd-42f6-b044-ce7069a322ae-istio-token\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:30:28.530643 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:28.530550    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/570f9db7-c9cd-42f6-b044-ce7069a322ae-istio-podinfo\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:30:28.530643 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:28.530563    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/570f9db7-c9cd-42f6-b044-ce7069a322ae-workload-certs\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:30:28.530643 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:28.530575    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/570f9db7-c9cd-42f6-b044-ce7069a322ae-credential-socket\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:30:28.530643 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:28.530588    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/570f9db7-c9cd-42f6-b044-ce7069a322ae-istio-envoy\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:30:28.533073 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:28.533050    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f-maas-api-tls\") pod \"maas-api-e2e-shared-a-677131-597975fd87-h7pjv\" (UID: \"8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-677131-597975fd87-h7pjv"
Jul 15 21:30:28.533611 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:28.533594    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-843ced-openshift-default-9764f7bcf-csdqh"]
Jul 15 21:30:28.708781 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:28.708741    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-677131-597975fd87-h7pjv"
Jul 15 21:30:28.851699 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:28.851532    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-677131-597975fd87-h7pjv"]
Jul 15 21:30:28.853791 ip-10-0-137-234 kubenswrapper[2566]: W0715 21:30:28.853766    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f4803f2_25e5_43aa_819b_6e5f6b3c7d6f.slice/crio-6803d529f4e4b5ca922f42c8f9c782d2bc5353d717618d8c87437cbe47c65e80 WatchSource:0}: Error finding container 6803d529f4e4b5ca922f42c8f9c782d2bc5353d717618d8c87437cbe47c65e80: Status 404 returned error can't find the container with id 6803d529f4e4b5ca922f42c8f9c782d2bc5353d717618d8c87437cbe47c65e80
Jul 15 21:30:29.509920 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:29.509865    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-677131-597975fd87-h7pjv" event={"ID":"8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f","Type":"ContainerStarted","Data":"6803d529f4e4b5ca922f42c8f9c782d2bc5353d717618d8c87437cbe47c65e80"}
Jul 15 21:30:29.756944 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:29.756910    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="570f9db7-c9cd-42f6-b044-ce7069a322ae" path="/var/lib/kubelet/pods/570f9db7-c9cd-42f6-b044-ce7069a322ae/volumes"
Jul 15 21:30:30.516643 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:30.516606    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-677131-597975fd87-h7pjv" event={"ID":"8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f","Type":"ContainerStarted","Data":"71e5a5ae0685a78d99660e385a5543cc74ee1cea2985cc68ddbe233be59c191c"}
Jul 15 21:30:30.516815 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:30.516669    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-677131-597975fd87-h7pjv"
Jul 15 21:30:30.539756 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:30.539667    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-677131-597975fd87-h7pjv" podStartSLOduration=2.120761765 podStartE2EDuration="3.53965131s" totalImagesPullingTime="1.418889545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:30:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:30:30.538303789 +0000 UTC m=+2563.687341865" watchObservedRunningTime="2026-07-15 21:30:30.53965131 +0000 UTC m=+2563.688689371"
Jul 15 21:30:32.051783 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:32.051732    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:30:32.425772 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:32.425660    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:30:32.733798 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:32.733765    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt"]
Jul 15 21:30:32.734306 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:32.734290    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="570f9db7-c9cd-42f6-b044-ce7069a322ae" containerName="istio-proxy"
Jul 15 21:30:32.734367 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:32.734309    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="570f9db7-c9cd-42f6-b044-ce7069a322ae" containerName="istio-proxy"
Jul 15 21:30:32.734367 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:32.734329    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="570f9db7-c9cd-42f6-b044-ce7069a322ae" containerName="istio-proxy"
Jul 15 21:30:32.734367 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:32.734337    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="570f9db7-c9cd-42f6-b044-ce7069a322ae" containerName="istio-proxy"
Jul 15 21:30:32.734519 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:32.734457    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="570f9db7-c9cd-42f6-b044-ce7069a322ae" containerName="istio-proxy"
Jul 15 21:30:32.734519 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:32.734471    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="570f9db7-c9cd-42f6-b044-ce7069a322ae" containerName="istio-proxy"
Jul 15 21:30:32.737963 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:32.737941    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt"
Jul 15 21:30:32.741275 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:32.741253    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-266bf8-openshift-default-dockercfg-7hgx7\""
Jul 15 21:30:32.754524 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:32.754501    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt"]
Jul 15 21:30:32.785517 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:32.785487    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:30:32.870752 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:32.870718    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45fa9c5b-9ffe-4367-9e24-18a48af5181c-istio-envoy\") pod \"e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt\" (UID: \"45fa9c5b-9ffe-4367-9e24-18a48af5181c\") " pod="openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt"
Jul 15 21:30:32.870967 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:32.870760    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45fa9c5b-9ffe-4367-9e24-18a48af5181c-istio-podinfo\") pod \"e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt\" (UID: \"45fa9c5b-9ffe-4367-9e24-18a48af5181c\") " pod="openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt"
Jul 15 21:30:32.870967 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:32.870813    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxk56\" (UniqueName: \"kubernetes.io/projected/45fa9c5b-9ffe-4367-9e24-18a48af5181c-kube-api-access-lxk56\") pod \"e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt\" (UID: \"45fa9c5b-9ffe-4367-9e24-18a48af5181c\") " pod="openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt"
Jul 15 21:30:32.870967 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:32.870850    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45fa9c5b-9ffe-4367-9e24-18a48af5181c-credential-socket\") pod \"e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt\" (UID: \"45fa9c5b-9ffe-4367-9e24-18a48af5181c\") " pod="openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt"
Jul 15 21:30:32.871148 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:32.871072    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45fa9c5b-9ffe-4367-9e24-18a48af5181c-istio-data\") pod \"e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt\" (UID: \"45fa9c5b-9ffe-4367-9e24-18a48af5181c\") " pod="openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt"
Jul 15 21:30:32.871148 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:32.871124    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45fa9c5b-9ffe-4367-9e24-18a48af5181c-workload-socket\") pod \"e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt\" (UID: \"45fa9c5b-9ffe-4367-9e24-18a48af5181c\") " pod="openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt"
Jul 15 21:30:32.871257 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:32.871148    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45fa9c5b-9ffe-4367-9e24-18a48af5181c-istio-token\") pod \"e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt\" (UID: \"45fa9c5b-9ffe-4367-9e24-18a48af5181c\") " pod="openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt"
Jul 15 21:30:32.871257 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:32.871178    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45fa9c5b-9ffe-4367-9e24-18a48af5181c-istiod-ca-cert\") pod \"e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt\" (UID: \"45fa9c5b-9ffe-4367-9e24-18a48af5181c\") " pod="openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt"
Jul 15 21:30:32.871257 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:32.871230    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45fa9c5b-9ffe-4367-9e24-18a48af5181c-workload-certs\") pod \"e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt\" (UID: \"45fa9c5b-9ffe-4367-9e24-18a48af5181c\") " pod="openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt"
Jul 15 21:30:32.972352 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:32.972312    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45fa9c5b-9ffe-4367-9e24-18a48af5181c-istio-data\") pod \"e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt\" (UID: \"45fa9c5b-9ffe-4367-9e24-18a48af5181c\") " pod="openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt"
Jul 15 21:30:32.972560 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:32.972362    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45fa9c5b-9ffe-4367-9e24-18a48af5181c-workload-socket\") pod \"e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt\" (UID: \"45fa9c5b-9ffe-4367-9e24-18a48af5181c\") " pod="openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt"
Jul 15 21:30:32.972560 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:32.972379    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45fa9c5b-9ffe-4367-9e24-18a48af5181c-istio-token\") pod \"e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt\" (UID: \"45fa9c5b-9ffe-4367-9e24-18a48af5181c\") " pod="openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt"
Jul 15 21:30:32.972560 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:32.972397    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45fa9c5b-9ffe-4367-9e24-18a48af5181c-istiod-ca-cert\") pod \"e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt\" (UID: \"45fa9c5b-9ffe-4367-9e24-18a48af5181c\") " pod="openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt"
Jul 15 21:30:32.972560 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:32.972428    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45fa9c5b-9ffe-4367-9e24-18a48af5181c-workload-certs\") pod \"e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt\" (UID: \"45fa9c5b-9ffe-4367-9e24-18a48af5181c\") " pod="openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt"
Jul 15 21:30:32.972560 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:32.972468    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45fa9c5b-9ffe-4367-9e24-18a48af5181c-istio-envoy\") pod \"e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt\" (UID: \"45fa9c5b-9ffe-4367-9e24-18a48af5181c\") " pod="openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt"
Jul 15 21:30:32.972560 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:32.972495    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45fa9c5b-9ffe-4367-9e24-18a48af5181c-istio-podinfo\") pod \"e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt\" (UID: \"45fa9c5b-9ffe-4367-9e24-18a48af5181c\") " pod="openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt"
Jul 15 21:30:32.972560 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:32.972525    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxk56\" (UniqueName: \"kubernetes.io/projected/45fa9c5b-9ffe-4367-9e24-18a48af5181c-kube-api-access-lxk56\") pod \"e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt\" (UID: \"45fa9c5b-9ffe-4367-9e24-18a48af5181c\") " pod="openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt"
Jul 15 21:30:32.972984 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:32.972660    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45fa9c5b-9ffe-4367-9e24-18a48af5181c-credential-socket\") pod \"e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt\" (UID: \"45fa9c5b-9ffe-4367-9e24-18a48af5181c\") " pod="openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt"
Jul 15 21:30:32.972984 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:32.972824    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45fa9c5b-9ffe-4367-9e24-18a48af5181c-workload-socket\") pod \"e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt\" (UID: \"45fa9c5b-9ffe-4367-9e24-18a48af5181c\") " pod="openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt"
Jul 15 21:30:32.972984 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:32.972923    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45fa9c5b-9ffe-4367-9e24-18a48af5181c-workload-certs\") pod \"e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt\" (UID: \"45fa9c5b-9ffe-4367-9e24-18a48af5181c\") " pod="openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt"
Jul 15 21:30:32.972984 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:32.972943    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45fa9c5b-9ffe-4367-9e24-18a48af5181c-istio-data\") pod \"e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt\" (UID: \"45fa9c5b-9ffe-4367-9e24-18a48af5181c\") " pod="openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt"
Jul 15 21:30:32.973133 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:32.973021    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45fa9c5b-9ffe-4367-9e24-18a48af5181c-credential-socket\") pod \"e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt\" (UID: \"45fa9c5b-9ffe-4367-9e24-18a48af5181c\") " pod="openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt"
Jul 15 21:30:32.973364 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:32.973336    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45fa9c5b-9ffe-4367-9e24-18a48af5181c-istiod-ca-cert\") pod \"e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt\" (UID: \"45fa9c5b-9ffe-4367-9e24-18a48af5181c\") " pod="openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt"
Jul 15 21:30:32.974965 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:32.974942    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45fa9c5b-9ffe-4367-9e24-18a48af5181c-istio-envoy\") pod \"e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt\" (UID: \"45fa9c5b-9ffe-4367-9e24-18a48af5181c\") " pod="openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt"
Jul 15 21:30:32.975090 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:32.975071    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45fa9c5b-9ffe-4367-9e24-18a48af5181c-istio-podinfo\") pod \"e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt\" (UID: \"45fa9c5b-9ffe-4367-9e24-18a48af5181c\") " pod="openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt"
Jul 15 21:30:32.982448 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:32.982417    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxk56\" (UniqueName: \"kubernetes.io/projected/45fa9c5b-9ffe-4367-9e24-18a48af5181c-kube-api-access-lxk56\") pod \"e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt\" (UID: \"45fa9c5b-9ffe-4367-9e24-18a48af5181c\") " pod="openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt"
Jul 15 21:30:32.982832 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:32.982811    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45fa9c5b-9ffe-4367-9e24-18a48af5181c-istio-token\") pod \"e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt\" (UID: \"45fa9c5b-9ffe-4367-9e24-18a48af5181c\") " pod="openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt"
Jul 15 21:30:33.049260 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:33.049172    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt"
Jul 15 21:30:33.173848 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:33.173793    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:30:33.195361 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:33.195334    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt"]
Jul 15 21:30:33.198061 ip-10-0-137-234 kubenswrapper[2566]: W0715 21:30:33.198017    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45fa9c5b_9ffe_4367_9e24_18a48af5181c.slice/crio-7cb58f865b5c9bcdfcd4b1662e86fe2f46dfecd088a40e1254ecc6ff6beb50ce WatchSource:0}: Error finding container 7cb58f865b5c9bcdfcd4b1662e86fe2f46dfecd088a40e1254ecc6ff6beb50ce: Status 404 returned error can't find the container with id 7cb58f865b5c9bcdfcd4b1662e86fe2f46dfecd088a40e1254ecc6ff6beb50ce
Jul 15 21:30:33.200874 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:33.200844    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:30:33.200984 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:33.200948    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:30:33.201030 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:33.200985    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:30:33.531822 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:33.531782    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt" event={"ID":"45fa9c5b-9ffe-4367-9e24-18a48af5181c","Type":"ContainerStarted","Data":"39be6624d44129ad83663b3d57bc44d60677234ed42e2ec52e498bc8d516b2aa"}
Jul 15 21:30:33.532049 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:33.531848    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt" event={"ID":"45fa9c5b-9ffe-4367-9e24-18a48af5181c","Type":"ContainerStarted","Data":"7cb58f865b5c9bcdfcd4b1662e86fe2f46dfecd088a40e1254ecc6ff6beb50ce"}
Jul 15 21:30:33.555609 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:33.555564    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt" podStartSLOduration=1.5555483749999999 podStartE2EDuration="1.555548375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:30:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:30:33.554873366 +0000 UTC m=+2566.703911443" watchObservedRunningTime="2026-07-15 21:30:33.555548375 +0000 UTC m=+2566.704586431"
Jul 15 21:30:34.050362 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:34.050327    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt"
Jul 15 21:30:34.055144 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:34.055120    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt"
Jul 15 21:30:34.537153 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:34.537120    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt"
Jul 15 21:30:34.538111 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:34.538094    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt"
Jul 15 21:30:35.568969 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:35.568941    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:30:35.681093 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:35.681062    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:30:35.742564 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:35.742531    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-266bf8-6f7b6b4f4f-2xgqq"]
Jul 15 21:30:35.746322 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:35.746303    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-266bf8-6f7b6b4f4f-2xgqq"
Jul 15 21:30:35.749625 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:35.749604    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-266bf8-serving-cert\""
Jul 15 21:30:35.762789 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:35.762763    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-266bf8-6f7b6b4f4f-2xgqq"]
Jul 15 21:30:35.909285 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:35.909186    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-266bf8-6f7b6b4f4f-2xgqq\" (UID: \"a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-266bf8-6f7b6b4f4f-2xgqq"
Jul 15 21:30:35.909441 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:35.909327    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftsjm\" (UniqueName: \"kubernetes.io/projected/a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd-kube-api-access-ftsjm\") pod \"maas-api-e2e-shared-b-266bf8-6f7b6b4f4f-2xgqq\" (UID: \"a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-266bf8-6f7b6b4f4f-2xgqq"
Jul 15 21:30:35.909441 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:35.909403    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd-maas-api-tls\") pod \"maas-api-e2e-shared-b-266bf8-6f7b6b4f4f-2xgqq\" (UID: \"a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-266bf8-6f7b6b4f4f-2xgqq"
Jul 15 21:30:36.010791 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:36.010751    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftsjm\" (UniqueName: \"kubernetes.io/projected/a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd-kube-api-access-ftsjm\") pod \"maas-api-e2e-shared-b-266bf8-6f7b6b4f4f-2xgqq\" (UID: \"a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-266bf8-6f7b6b4f4f-2xgqq"
Jul 15 21:30:36.010997 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:36.010814    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd-maas-api-tls\") pod \"maas-api-e2e-shared-b-266bf8-6f7b6b4f4f-2xgqq\" (UID: \"a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-266bf8-6f7b6b4f4f-2xgqq"
Jul 15 21:30:36.010997 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:36.010873    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-266bf8-6f7b6b4f4f-2xgqq\" (UID: \"a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-266bf8-6f7b6b4f4f-2xgqq"
Jul 15 21:30:36.011112 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:30:36.011004    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-266bf8-serving-cert: secret "maas-api-e2e-shared-b-266bf8-serving-cert" not found
Jul 15 21:30:36.011112 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:30:36.011070    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd-maas-api-tls podName:a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd nodeName:}" failed. No retries permitted until 2026-07-15 21:30:36.51105409 +0000 UTC m=+2569.660092146 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd-maas-api-tls") pod "maas-api-e2e-shared-b-266bf8-6f7b6b4f4f-2xgqq" (UID: "a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd") : secret "maas-api-e2e-shared-b-266bf8-serving-cert" not found
Jul 15 21:30:36.011553 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:36.011535    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-266bf8-6f7b6b4f4f-2xgqq\" (UID: \"a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-266bf8-6f7b6b4f4f-2xgqq"
Jul 15 21:30:36.030553 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:36.030516    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftsjm\" (UniqueName: \"kubernetes.io/projected/a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd-kube-api-access-ftsjm\") pod \"maas-api-e2e-shared-b-266bf8-6f7b6b4f4f-2xgqq\" (UID: \"a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-266bf8-6f7b6b4f4f-2xgqq"
Jul 15 21:30:36.131506 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:36.131465    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:30:36.407305 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:36.407218    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:30:36.506182 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:36.506064    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:30:36.516121 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:36.516087    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd-maas-api-tls\") pod \"maas-api-e2e-shared-b-266bf8-6f7b6b4f4f-2xgqq\" (UID: \"a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-266bf8-6f7b6b4f4f-2xgqq"
Jul 15 21:30:36.516255 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:30:36.516214    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-266bf8-serving-cert: secret "maas-api-e2e-shared-b-266bf8-serving-cert" not found
Jul 15 21:30:36.516295 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:30:36.516272    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd-maas-api-tls podName:a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd nodeName:}" failed. No retries permitted until 2026-07-15 21:30:37.516258716 +0000 UTC m=+2570.665296754 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd-maas-api-tls") pod "maas-api-e2e-shared-b-266bf8-6f7b6b4f4f-2xgqq" (UID: "a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd") : secret "maas-api-e2e-shared-b-266bf8-serving-cert" not found
Jul 15 21:30:36.528119 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:36.528088    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-677131-597975fd87-h7pjv"
Jul 15 21:30:36.729854 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:36.729829    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:30:37.525250 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:37.525217    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd-maas-api-tls\") pod \"maas-api-e2e-shared-b-266bf8-6f7b6b4f4f-2xgqq\" (UID: \"a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-266bf8-6f7b6b4f4f-2xgqq"
Jul 15 21:30:37.527843 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:37.527816    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd-maas-api-tls\") pod \"maas-api-e2e-shared-b-266bf8-6f7b6b4f4f-2xgqq\" (UID: \"a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-266bf8-6f7b6b4f4f-2xgqq"
Jul 15 21:30:37.557304 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:37.557269    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-266bf8-6f7b6b4f4f-2xgqq"
Jul 15 21:30:37.907810 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:37.907775    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-266bf8-6f7b6b4f4f-2xgqq"]
Jul 15 21:30:37.909051 ip-10-0-137-234 kubenswrapper[2566]: W0715 21:30:37.909024    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0ee92b9_f3af_4ebe_8603_70ca3c3e43fd.slice/crio-ed50462eb0f5d2fca431bd88f6dca3d897d92dfe4e9283ded339bfc9bc3e83b3 WatchSource:0}: Error finding container ed50462eb0f5d2fca431bd88f6dca3d897d92dfe4e9283ded339bfc9bc3e83b3: Status 404 returned error can't find the container with id ed50462eb0f5d2fca431bd88f6dca3d897d92dfe4e9283ded339bfc9bc3e83b3
Jul 15 21:30:38.567953 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:38.567918    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-266bf8-6f7b6b4f4f-2xgqq" event={"ID":"a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd","Type":"ContainerStarted","Data":"ed50462eb0f5d2fca431bd88f6dca3d897d92dfe4e9283ded339bfc9bc3e83b3"}
Jul 15 21:30:39.574520 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:39.574478    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-266bf8-6f7b6b4f4f-2xgqq" event={"ID":"a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd","Type":"ContainerStarted","Data":"cf9fa310d60226c4636c934fbb5b66948a71207ef431194d56ca8de50cb224a5"}
Jul 15 21:30:39.575021 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:39.574607    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-266bf8-6f7b6b4f4f-2xgqq"
Jul 15 21:30:39.596313 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:39.596260    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-266bf8-6f7b6b4f4f-2xgqq" podStartSLOduration=3.954802868 podStartE2EDuration="4.596242882s" totalImagesPullingTime="641.440014ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:30:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:30:39.594579929 +0000 UTC m=+2572.743617988" watchObservedRunningTime="2026-07-15 21:30:39.596242882 +0000 UTC m=+2572.745280968"
Jul 15 21:30:40.109319 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:40.109272    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:30:40.585512 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:40.583825    2566 generic.go:356] "Generic (PLEG): container finished" podID="a48bab9d-061c-45b7-8b8a-8e343d64058a" containerID="a8bdbdde8a0aeabfa114107ece61bcd6be780de2889913a700da6d69c09de98a" exitCode=0
Jul 15 21:30:40.585512 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:40.584531    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp" event={"ID":"a48bab9d-061c-45b7-8b8a-8e343d64058a","Type":"ContainerDied","Data":"a8bdbdde8a0aeabfa114107ece61bcd6be780de2889913a700da6d69c09de98a"}
Jul 15 21:30:40.585512 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:40.584940    2566 scope.go:117] "RemoveContainer" containerID="a8bdbdde8a0aeabfa114107ece61bcd6be780de2889913a700da6d69c09de98a"
Jul 15 21:30:40.588534 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:40.588497    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:30:40.593339 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:40.593310    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:30:40.593553 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:40.593530    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:30:41.491111 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:41.491065    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp"
Jul 15 21:30:41.594774 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:41.594727    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp" event={"ID":"a48bab9d-061c-45b7-8b8a-8e343d64058a","Type":"ContainerStarted","Data":"29ee16227685601282c8a3ca29b5603f052b716296c84c514af0c8dcb6f20e6f"}
Jul 15 21:30:42.353090 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:42.353047    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp"
Jul 15 21:30:42.354289 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:42.354265    2566 patch_prober.go:28] interesting pod/e2e-shared-a-677131-openshift-default-784d56455-tgbzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Jul 15 21:30:42.354348 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:42.354331    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp" podUID="a48bab9d-061c-45b7-8b8a-8e343d64058a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Jul 15 21:30:43.352484 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:43.352448    2566 patch_prober.go:28] interesting pod/e2e-shared-a-677131-openshift-default-784d56455-tgbzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Jul 15 21:30:43.352929 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:43.352509    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp" podUID="a48bab9d-061c-45b7-8b8a-8e343d64058a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Jul 15 21:30:44.352672 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:44.352638    2566 patch_prober.go:28] interesting pod/e2e-shared-a-677131-openshift-default-784d56455-tgbzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Jul 15 21:30:44.353067 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:44.352692    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp" podUID="a48bab9d-061c-45b7-8b8a-8e343d64058a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Jul 15 21:30:45.353372 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:45.353320    2566 patch_prober.go:28] interesting pod/e2e-shared-a-677131-openshift-default-784d56455-tgbzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Jul 15 21:30:45.353928 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:45.353391    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp" podUID="a48bab9d-061c-45b7-8b8a-8e343d64058a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Jul 15 21:30:45.590440 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:45.590407    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-266bf8-6f7b6b4f4f-2xgqq"
Jul 15 21:30:46.357705 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:46.357670    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp"
Jul 15 21:30:46.358196 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:46.358085    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp"
Jul 15 21:30:46.358881 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:46.358860    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp"
Jul 15 21:30:47.639870 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:47.639777    2566 generic.go:356] "Generic (PLEG): container finished" podID="45fa9c5b-9ffe-4367-9e24-18a48af5181c" containerID="39be6624d44129ad83663b3d57bc44d60677234ed42e2ec52e498bc8d516b2aa" exitCode=0
Jul 15 21:30:47.639870 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:47.639848    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt" event={"ID":"45fa9c5b-9ffe-4367-9e24-18a48af5181c","Type":"ContainerDied","Data":"39be6624d44129ad83663b3d57bc44d60677234ed42e2ec52e498bc8d516b2aa"}
Jul 15 21:30:47.640454 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:47.640435    2566 scope.go:117] "RemoveContainer" containerID="39be6624d44129ad83663b3d57bc44d60677234ed42e2ec52e498bc8d516b2aa"
Jul 15 21:30:47.641418 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:47.641394    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:30:47.641477 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:47.641452    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:30:47.641515 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:47.641482    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 15 21:30:48.646444 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:48.646407    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt" event={"ID":"45fa9c5b-9ffe-4367-9e24-18a48af5181c","Type":"ContainerStarted","Data":"eb48e463a778b1f2f46bf9f565aa48e5e88bd94ac96bafe1fcbba529de9faf9f"}
Jul 15 21:30:49.049954 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:49.049928    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt"
Jul 15 21:30:49.051077 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:49.051054    2566 patch_prober.go:28] interesting pod/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Jul 15 21:30:49.051146 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:49.051116    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt" podUID="45fa9c5b-9ffe-4367-9e24-18a48af5181c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Jul 15 21:30:49.537193 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:49.537158    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt"
Jul 15 21:30:50.049816 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:50.049780    2566 patch_prober.go:28] interesting pod/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Jul 15 21:30:50.050248 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:50.049842    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt" podUID="45fa9c5b-9ffe-4367-9e24-18a48af5181c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Jul 15 21:30:50.350408 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:50.350316    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:30:50.473422 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:50.473373    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:30:51.049745 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:51.049707    2566 patch_prober.go:28] interesting pod/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Jul 15 21:30:51.049945 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:51.049769    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt" podUID="45fa9c5b-9ffe-4367-9e24-18a48af5181c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Jul 15 21:30:52.050443 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:52.050407    2566 patch_prober.go:28] interesting pod/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Jul 15 21:30:52.050851 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:52.050471    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt" podUID="45fa9c5b-9ffe-4367-9e24-18a48af5181c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Jul 15 21:30:53.053557 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:53.053522    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt"
Jul 15 21:30:53.054472 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:53.054452    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt"
Jul 15 21:30:56.796691 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:56.796656    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:30:57.417080 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:57.417048    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:30:57.514729 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:30:57.514691    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:31:01.186317 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:31:01.186272    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:31:02.212689 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:31:02.212652    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:31:26.189928 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:31:26.189837    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:32:47.863155 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:32:47.863072    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-652lk_f9a159bb-39c4-4cca-92ff-1c74d739e311/ovn-acl-logging/0.log"
Jul 15 21:32:47.866170 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:32:47.866146    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-652lk_f9a159bb-39c4-4cca-92ff-1c74d739e311/ovn-acl-logging/0.log"
Jul 15 21:32:58.356088 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:32:58.356056    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:32:58.817131 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:32:58.817089    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:33:14.266196 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:33:14.266155    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:33:18.866219 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:33:18.866186    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:33:41.037665 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:33:41.037631    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:33:42.147602 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:33:42.147570    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:33:53.410581 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:33:53.410539    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:34:22.627860 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:34:22.627777    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:35:29.911101 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:35:29.911067    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:35:32.444342 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:35:32.444305    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:35:45.587087 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:35:45.587047    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:35:51.314633 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:35:51.314543    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:35:51.413592 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:35:51.413556    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:35:58.326285 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:35:58.326239    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:35:58.427303 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:35:58.427268    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:36:06.824117 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:36:06.824084    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:36:43.910072 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:36:43.910035    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:36:44.010651 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:36:44.010612    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:36:48.194928 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:36:48.194883    2566 scope.go:117] "RemoveContainer" containerID="58526e7b647bc5cdb2faf3e2c709054147ba2c5446f8a03fba0c6e7221a9ec18"
Jul 15 21:36:58.543790 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:36:58.543748    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:36:59.823266 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:36:59.823232    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:37:08.075278 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:08.075195    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:37:19.249043 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:19.249009    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:37:19.345666 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:19.345629    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:37:29.150753 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:29.150714    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-677131-597975fd87-h7pjv"]
Jul 15 21:37:29.151191 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:29.150985    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-677131-597975fd87-h7pjv" podUID="8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f" containerName="maas-api" containerID="cri-o://71e5a5ae0685a78d99660e385a5543cc74ee1cea2985cc68ddbe233be59c191c" gracePeriod=30
Jul 15 21:37:29.173762 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:29.173706    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:37:29.406558 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:29.406492    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-677131-597975fd87-h7pjv"
Jul 15 21:37:29.560341 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:29.560309    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bq888\" (UniqueName: \"kubernetes.io/projected/8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f-kube-api-access-bq888\") pod \"8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f\" (UID: \"8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f\") "
Jul 15 21:37:29.560341 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:29.560351    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f-openshift-service-ca\") pod \"8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f\" (UID: \"8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f\") "
Jul 15 21:37:29.560580 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:29.560444    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f-maas-api-tls\") pod \"8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f\" (UID: \"8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f\") "
Jul 15 21:37:29.560801 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:29.560773    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f" (UID: "8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 21:37:29.562401 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:29.562372    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f-kube-api-access-bq888" (OuterVolumeSpecName: "kube-api-access-bq888") pod "8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f" (UID: "8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f"). InnerVolumeSpecName "kube-api-access-bq888". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 21:37:29.562498 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:29.562472    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f" (UID: "8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 21:37:29.661319 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:29.661225    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq888\" (UniqueName: \"kubernetes.io/projected/8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f-kube-api-access-bq888\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:37:29.661319 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:29.661252    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f-openshift-service-ca\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:37:29.661319 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:29.661263    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f-maas-api-tls\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:37:29.665390 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:29.665360    2566 generic.go:356] "Generic (PLEG): container finished" podID="8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f" containerID="71e5a5ae0685a78d99660e385a5543cc74ee1cea2985cc68ddbe233be59c191c" exitCode=0
Jul 15 21:37:29.665515 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:29.665406    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-677131-597975fd87-h7pjv"
Jul 15 21:37:29.665515 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:29.665440    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-677131-597975fd87-h7pjv" event={"ID":"8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f","Type":"ContainerDied","Data":"71e5a5ae0685a78d99660e385a5543cc74ee1cea2985cc68ddbe233be59c191c"}
Jul 15 21:37:29.665515 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:29.665494    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-677131-597975fd87-h7pjv" event={"ID":"8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f","Type":"ContainerDied","Data":"6803d529f4e4b5ca922f42c8f9c782d2bc5353d717618d8c87437cbe47c65e80"}
Jul 15 21:37:29.665627 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:29.665518    2566 scope.go:117] "RemoveContainer" containerID="71e5a5ae0685a78d99660e385a5543cc74ee1cea2985cc68ddbe233be59c191c"
Jul 15 21:37:29.674693 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:29.674673    2566 scope.go:117] "RemoveContainer" containerID="71e5a5ae0685a78d99660e385a5543cc74ee1cea2985cc68ddbe233be59c191c"
Jul 15 21:37:29.675003 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:37:29.674974    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71e5a5ae0685a78d99660e385a5543cc74ee1cea2985cc68ddbe233be59c191c\": container with ID starting with 71e5a5ae0685a78d99660e385a5543cc74ee1cea2985cc68ddbe233be59c191c not found: ID does not exist" containerID="71e5a5ae0685a78d99660e385a5543cc74ee1cea2985cc68ddbe233be59c191c"
Jul 15 21:37:29.675105 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:29.675012    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71e5a5ae0685a78d99660e385a5543cc74ee1cea2985cc68ddbe233be59c191c"} err="failed to get container status \"71e5a5ae0685a78d99660e385a5543cc74ee1cea2985cc68ddbe233be59c191c\": rpc error: code = NotFound desc = could not find container \"71e5a5ae0685a78d99660e385a5543cc74ee1cea2985cc68ddbe233be59c191c\": container with ID starting with 71e5a5ae0685a78d99660e385a5543cc74ee1cea2985cc68ddbe233be59c191c not found: ID does not exist"
Jul 15 21:37:29.689424 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:29.689394    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-677131-597975fd87-h7pjv"]
Jul 15 21:37:29.692500 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:29.692477    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-677131-597975fd87-h7pjv"]
Jul 15 21:37:29.756585 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:29.756556    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f" path="/var/lib/kubelet/pods/8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f/volumes"
Jul 15 21:37:33.804673 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:33.804616    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:37:40.268589 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:40.268556    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:37:40.386865 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:40.386832    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:37:44.478379 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:44.478341    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp"]
Jul 15 21:37:44.478810 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:44.478651    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp" podUID="a48bab9d-061c-45b7-8b8a-8e343d64058a" containerName="istio-proxy" containerID="cri-o://29ee16227685601282c8a3ca29b5603f052b716296c84c514af0c8dcb6f20e6f" gracePeriod=30
Jul 15 21:37:44.501592 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:44.501558    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:37:44.751037 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:44.750915    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:37:45.459581 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:45.459543    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:37:45.559856 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:45.559816    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:37:46.360612 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:46.360578    2566 patch_prober.go:28] interesting pod/e2e-shared-a-677131-openshift-default-784d56455-tgbzp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 15 21:37:46.360784 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:46.360636    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp" podUID="a48bab9d-061c-45b7-8b8a-8e343d64058a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 15 21:37:47.893964 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:47.893935    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-652lk_f9a159bb-39c4-4cca-92ff-1c74d739e311/ovn-acl-logging/0.log"
Jul 15 21:37:47.897570 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:47.897546    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-652lk_f9a159bb-39c4-4cca-92ff-1c74d739e311/ovn-acl-logging/0.log"
Jul 15 21:37:49.738120 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:49.738097    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp"
Jul 15 21:37:49.771007 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:49.770932    2566 generic.go:356] "Generic (PLEG): container finished" podID="a48bab9d-061c-45b7-8b8a-8e343d64058a" containerID="29ee16227685601282c8a3ca29b5603f052b716296c84c514af0c8dcb6f20e6f" exitCode=0
Jul 15 21:37:49.771007 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:49.770980    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp" event={"ID":"a48bab9d-061c-45b7-8b8a-8e343d64058a","Type":"ContainerDied","Data":"29ee16227685601282c8a3ca29b5603f052b716296c84c514af0c8dcb6f20e6f"}
Jul 15 21:37:49.771007 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:49.770996    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp"
Jul 15 21:37:49.771235 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:49.771026    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp" event={"ID":"a48bab9d-061c-45b7-8b8a-8e343d64058a","Type":"ContainerDied","Data":"3e73a9eaa12c26a06767142a515734dac50f09e2f2fef77b699fc2030ea8611b"}
Jul 15 21:37:49.771235 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:49.771050    2566 scope.go:117] "RemoveContainer" containerID="29ee16227685601282c8a3ca29b5603f052b716296c84c514af0c8dcb6f20e6f"
Jul 15 21:37:49.779977 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:49.779952    2566 scope.go:117] "RemoveContainer" containerID="a8bdbdde8a0aeabfa114107ece61bcd6be780de2889913a700da6d69c09de98a"
Jul 15 21:37:49.787773 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:49.787754    2566 scope.go:117] "RemoveContainer" containerID="29ee16227685601282c8a3ca29b5603f052b716296c84c514af0c8dcb6f20e6f"
Jul 15 21:37:49.788062 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:37:49.788040    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29ee16227685601282c8a3ca29b5603f052b716296c84c514af0c8dcb6f20e6f\": container with ID starting with 29ee16227685601282c8a3ca29b5603f052b716296c84c514af0c8dcb6f20e6f not found: ID does not exist" containerID="29ee16227685601282c8a3ca29b5603f052b716296c84c514af0c8dcb6f20e6f"
Jul 15 21:37:49.788148 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:49.788073    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29ee16227685601282c8a3ca29b5603f052b716296c84c514af0c8dcb6f20e6f"} err="failed to get container status \"29ee16227685601282c8a3ca29b5603f052b716296c84c514af0c8dcb6f20e6f\": rpc error: code = NotFound desc = could not find container \"29ee16227685601282c8a3ca29b5603f052b716296c84c514af0c8dcb6f20e6f\": container with ID starting with 29ee16227685601282c8a3ca29b5603f052b716296c84c514af0c8dcb6f20e6f not found: ID does not exist"
Jul 15 21:37:49.788148 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:49.788100    2566 scope.go:117] "RemoveContainer" containerID="a8bdbdde8a0aeabfa114107ece61bcd6be780de2889913a700da6d69c09de98a"
Jul 15 21:37:49.788375 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:37:49.788357    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8bdbdde8a0aeabfa114107ece61bcd6be780de2889913a700da6d69c09de98a\": container with ID starting with a8bdbdde8a0aeabfa114107ece61bcd6be780de2889913a700da6d69c09de98a not found: ID does not exist" containerID="a8bdbdde8a0aeabfa114107ece61bcd6be780de2889913a700da6d69c09de98a"
Jul 15 21:37:49.788419 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:49.788382    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8bdbdde8a0aeabfa114107ece61bcd6be780de2889913a700da6d69c09de98a"} err="failed to get container status \"a8bdbdde8a0aeabfa114107ece61bcd6be780de2889913a700da6d69c09de98a\": rpc error: code = NotFound desc = could not find container \"a8bdbdde8a0aeabfa114107ece61bcd6be780de2889913a700da6d69c09de98a\": container with ID starting with a8bdbdde8a0aeabfa114107ece61bcd6be780de2889913a700da6d69c09de98a not found: ID does not exist"
Jul 15 21:37:49.828411 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:49.828383    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a48bab9d-061c-45b7-8b8a-8e343d64058a-istio-token\") pod \"a48bab9d-061c-45b7-8b8a-8e343d64058a\" (UID: \"a48bab9d-061c-45b7-8b8a-8e343d64058a\") "
Jul 15 21:37:49.828545 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:49.828439    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a48bab9d-061c-45b7-8b8a-8e343d64058a-istio-envoy\") pod \"a48bab9d-061c-45b7-8b8a-8e343d64058a\" (UID: \"a48bab9d-061c-45b7-8b8a-8e343d64058a\") "
Jul 15 21:37:49.828545 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:49.828456    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m2jgg\" (UniqueName: \"kubernetes.io/projected/a48bab9d-061c-45b7-8b8a-8e343d64058a-kube-api-access-m2jgg\") pod \"a48bab9d-061c-45b7-8b8a-8e343d64058a\" (UID: \"a48bab9d-061c-45b7-8b8a-8e343d64058a\") "
Jul 15 21:37:49.828545 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:49.828491    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a48bab9d-061c-45b7-8b8a-8e343d64058a-istio-data\") pod \"a48bab9d-061c-45b7-8b8a-8e343d64058a\" (UID: \"a48bab9d-061c-45b7-8b8a-8e343d64058a\") "
Jul 15 21:37:49.828545 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:49.828519    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a48bab9d-061c-45b7-8b8a-8e343d64058a-credential-socket\") pod \"a48bab9d-061c-45b7-8b8a-8e343d64058a\" (UID: \"a48bab9d-061c-45b7-8b8a-8e343d64058a\") "
Jul 15 21:37:49.828545 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:49.828538    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a48bab9d-061c-45b7-8b8a-8e343d64058a-workload-certs\") pod \"a48bab9d-061c-45b7-8b8a-8e343d64058a\" (UID: \"a48bab9d-061c-45b7-8b8a-8e343d64058a\") "
Jul 15 21:37:49.828871 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:49.828848    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a48bab9d-061c-45b7-8b8a-8e343d64058a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a48bab9d-061c-45b7-8b8a-8e343d64058a" (UID: "a48bab9d-061c-45b7-8b8a-8e343d64058a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:37:49.828871 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:49.828860    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a48bab9d-061c-45b7-8b8a-8e343d64058a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a48bab9d-061c-45b7-8b8a-8e343d64058a" (UID: "a48bab9d-061c-45b7-8b8a-8e343d64058a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:37:49.829007 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:49.828928    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a48bab9d-061c-45b7-8b8a-8e343d64058a-istio-podinfo\") pod \"a48bab9d-061c-45b7-8b8a-8e343d64058a\" (UID: \"a48bab9d-061c-45b7-8b8a-8e343d64058a\") "
Jul 15 21:37:49.829007 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:49.828985    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a48bab9d-061c-45b7-8b8a-8e343d64058a-workload-socket\") pod \"a48bab9d-061c-45b7-8b8a-8e343d64058a\" (UID: \"a48bab9d-061c-45b7-8b8a-8e343d64058a\") "
Jul 15 21:37:49.829114 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:49.829020    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a48bab9d-061c-45b7-8b8a-8e343d64058a-istiod-ca-cert\") pod \"a48bab9d-061c-45b7-8b8a-8e343d64058a\" (UID: \"a48bab9d-061c-45b7-8b8a-8e343d64058a\") "
Jul 15 21:37:49.829342 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:49.829287    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a48bab9d-061c-45b7-8b8a-8e343d64058a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a48bab9d-061c-45b7-8b8a-8e343d64058a" (UID: "a48bab9d-061c-45b7-8b8a-8e343d64058a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:37:49.829509 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:49.829375    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a48bab9d-061c-45b7-8b8a-8e343d64058a-credential-socket\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:37:49.829509 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:49.829395    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a48bab9d-061c-45b7-8b8a-8e343d64058a-workload-certs\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:37:49.829509 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:49.829425    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a48bab9d-061c-45b7-8b8a-8e343d64058a-workload-socket\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:37:49.829708 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:49.829510    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a48bab9d-061c-45b7-8b8a-8e343d64058a-istio-data" (OuterVolumeSpecName: "istio-data") pod "a48bab9d-061c-45b7-8b8a-8e343d64058a" (UID: "a48bab9d-061c-45b7-8b8a-8e343d64058a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:37:49.829708 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:49.829595    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a48bab9d-061c-45b7-8b8a-8e343d64058a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a48bab9d-061c-45b7-8b8a-8e343d64058a" (UID: "a48bab9d-061c-45b7-8b8a-8e343d64058a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 21:37:49.830818 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:49.830797    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a48bab9d-061c-45b7-8b8a-8e343d64058a-istio-token" (OuterVolumeSpecName: "istio-token") pod "a48bab9d-061c-45b7-8b8a-8e343d64058a" (UID: "a48bab9d-061c-45b7-8b8a-8e343d64058a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 21:37:49.830818 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:49.830812    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a48bab9d-061c-45b7-8b8a-8e343d64058a-kube-api-access-m2jgg" (OuterVolumeSpecName: "kube-api-access-m2jgg") pod "a48bab9d-061c-45b7-8b8a-8e343d64058a" (UID: "a48bab9d-061c-45b7-8b8a-8e343d64058a"). InnerVolumeSpecName "kube-api-access-m2jgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 21:37:49.830974 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:49.830814    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a48bab9d-061c-45b7-8b8a-8e343d64058a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a48bab9d-061c-45b7-8b8a-8e343d64058a" (UID: "a48bab9d-061c-45b7-8b8a-8e343d64058a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:37:49.830974 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:49.830964    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a48bab9d-061c-45b7-8b8a-8e343d64058a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a48bab9d-061c-45b7-8b8a-8e343d64058a" (UID: "a48bab9d-061c-45b7-8b8a-8e343d64058a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 15 21:37:49.930120 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:49.930096    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a48bab9d-061c-45b7-8b8a-8e343d64058a-istio-envoy\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:37:49.930120 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:49.930119    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2jgg\" (UniqueName: \"kubernetes.io/projected/a48bab9d-061c-45b7-8b8a-8e343d64058a-kube-api-access-m2jgg\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:37:49.930272 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:49.930130    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a48bab9d-061c-45b7-8b8a-8e343d64058a-istio-data\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:37:49.930272 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:49.930138    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a48bab9d-061c-45b7-8b8a-8e343d64058a-istio-podinfo\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:37:49.930272 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:49.930146    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a48bab9d-061c-45b7-8b8a-8e343d64058a-istiod-ca-cert\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:37:49.930272 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:49.930156    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a48bab9d-061c-45b7-8b8a-8e343d64058a-istio-token\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:37:50.095042 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:50.095008    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp"]
Jul 15 21:37:50.101963 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:50.101935    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-677131-openshift-default-784d56455-tgbzp"]
Jul 15 21:37:51.756238 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:51.756197    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a48bab9d-061c-45b7-8b8a-8e343d64058a" path="/var/lib/kubelet/pods/a48bab9d-061c-45b7-8b8a-8e343d64058a/volumes"
Jul 15 21:37:55.437455 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:55.437418    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-266bf8-6f7b6b4f4f-2xgqq"]
Jul 15 21:37:55.437884 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:55.437705    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-266bf8-6f7b6b4f4f-2xgqq" podUID="a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd" containerName="maas-api" containerID="cri-o://cf9fa310d60226c4636c934fbb5b66948a71207ef431194d56ca8de50cb224a5" gracePeriod=30
Jul 15 21:37:55.457438 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:55.457378    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:37:55.585806 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:55.585765    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-266bf8-6f7b6b4f4f-2xgqq" podUID="a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.67:8443/health\": dial tcp 10.134.0.67:8443: connect: connection refused"
Jul 15 21:37:55.689620 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:55.689554    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-266bf8-6f7b6b4f4f-2xgqq"
Jul 15 21:37:55.781341 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:55.781303    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd-openshift-service-ca\") pod \"a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd\" (UID: \"a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd\") "
Jul 15 21:37:55.781527 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:55.781383    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd-maas-api-tls\") pod \"a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd\" (UID: \"a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd\") "
Jul 15 21:37:55.781527 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:55.781412    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ftsjm\" (UniqueName: \"kubernetes.io/projected/a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd-kube-api-access-ftsjm\") pod \"a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd\" (UID: \"a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd\") "
Jul 15 21:37:55.781720 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:55.781697    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd" (UID: "a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 21:37:55.783534 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:55.783505    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd" (UID: "a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 21:37:55.783907 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:55.783872    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd-kube-api-access-ftsjm" (OuterVolumeSpecName: "kube-api-access-ftsjm") pod "a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd" (UID: "a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd"). InnerVolumeSpecName "kube-api-access-ftsjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 21:37:55.800347 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:55.800314    2566 generic.go:356] "Generic (PLEG): container finished" podID="a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd" containerID="cf9fa310d60226c4636c934fbb5b66948a71207ef431194d56ca8de50cb224a5" exitCode=0
Jul 15 21:37:55.800482 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:55.800360    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-266bf8-6f7b6b4f4f-2xgqq" event={"ID":"a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd","Type":"ContainerDied","Data":"cf9fa310d60226c4636c934fbb5b66948a71207ef431194d56ca8de50cb224a5"}
Jul 15 21:37:55.800482 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:55.800367    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-266bf8-6f7b6b4f4f-2xgqq"
Jul 15 21:37:55.800482 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:55.800403    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-266bf8-6f7b6b4f4f-2xgqq" event={"ID":"a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd","Type":"ContainerDied","Data":"ed50462eb0f5d2fca431bd88f6dca3d897d92dfe4e9283ded339bfc9bc3e83b3"}
Jul 15 21:37:55.800482 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:55.800424    2566 scope.go:117] "RemoveContainer" containerID="cf9fa310d60226c4636c934fbb5b66948a71207ef431194d56ca8de50cb224a5"
Jul 15 21:37:55.809393 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:55.809375    2566 scope.go:117] "RemoveContainer" containerID="cf9fa310d60226c4636c934fbb5b66948a71207ef431194d56ca8de50cb224a5"
Jul 15 21:37:55.809667 ip-10-0-137-234 kubenswrapper[2566]: E0715 21:37:55.809646    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf9fa310d60226c4636c934fbb5b66948a71207ef431194d56ca8de50cb224a5\": container with ID starting with cf9fa310d60226c4636c934fbb5b66948a71207ef431194d56ca8de50cb224a5 not found: ID does not exist" containerID="cf9fa310d60226c4636c934fbb5b66948a71207ef431194d56ca8de50cb224a5"
Jul 15 21:37:55.809710 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:55.809675    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf9fa310d60226c4636c934fbb5b66948a71207ef431194d56ca8de50cb224a5"} err="failed to get container status \"cf9fa310d60226c4636c934fbb5b66948a71207ef431194d56ca8de50cb224a5\": rpc error: code = NotFound desc = could not find container \"cf9fa310d60226c4636c934fbb5b66948a71207ef431194d56ca8de50cb224a5\": container with ID starting with cf9fa310d60226c4636c934fbb5b66948a71207ef431194d56ca8de50cb224a5 not found: ID does not exist"
Jul 15 21:37:55.823796 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:55.823739    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-266bf8-6f7b6b4f4f-2xgqq"]
Jul 15 21:37:55.826227 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:55.826203    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-266bf8-6f7b6b4f4f-2xgqq"]
Jul 15 21:37:55.882944 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:55.882910    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftsjm\" (UniqueName: \"kubernetes.io/projected/a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd-kube-api-access-ftsjm\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:37:55.882944 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:55.882944    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd-openshift-service-ca\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:37:55.883154 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:55.882959    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd-maas-api-tls\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:37:57.756720 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:57.756685    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd" path="/var/lib/kubelet/pods/a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd/volumes"
Jul 15 21:37:59.173589 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:59.173551    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:37:59.274443 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:59.274402    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:37:59.966588 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:37:59.966543    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:38:06.675714 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:06.675671    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:38:06.779205 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:06.779174    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:38:10.684079 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:10.684037    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt"]
Jul 15 21:38:10.684525 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:10.684295    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt" podUID="45fa9c5b-9ffe-4367-9e24-18a48af5181c" containerName="istio-proxy" containerID="cri-o://eb48e463a778b1f2f46bf9f565aa48e5e88bd94ac96bafe1fcbba529de9faf9f" gracePeriod=30
Jul 15 21:38:10.700131 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:10.700100    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:38:10.940338 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:10.940244    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:38:15.898516 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:15.898479    2566 generic.go:356] "Generic (PLEG): container finished" podID="45fa9c5b-9ffe-4367-9e24-18a48af5181c" containerID="eb48e463a778b1f2f46bf9f565aa48e5e88bd94ac96bafe1fcbba529de9faf9f" exitCode=0
Jul 15 21:38:15.899077 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:15.898516    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt" event={"ID":"45fa9c5b-9ffe-4367-9e24-18a48af5181c","Type":"ContainerDied","Data":"eb48e463a778b1f2f46bf9f565aa48e5e88bd94ac96bafe1fcbba529de9faf9f"}
Jul 15 21:38:15.899077 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:15.898582    2566 scope.go:117] "RemoveContainer" containerID="39be6624d44129ad83663b3d57bc44d60677234ed42e2ec52e498bc8d516b2aa"
Jul 15 21:38:15.943346 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:15.943324    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt"
Jul 15 21:38:16.053781 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:16.053700    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45fa9c5b-9ffe-4367-9e24-18a48af5181c-istio-envoy\") pod \"45fa9c5b-9ffe-4367-9e24-18a48af5181c\" (UID: \"45fa9c5b-9ffe-4367-9e24-18a48af5181c\") "
Jul 15 21:38:16.053781 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:16.053735    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45fa9c5b-9ffe-4367-9e24-18a48af5181c-istiod-ca-cert\") pod \"45fa9c5b-9ffe-4367-9e24-18a48af5181c\" (UID: \"45fa9c5b-9ffe-4367-9e24-18a48af5181c\") "
Jul 15 21:38:16.053781 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:16.053763    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45fa9c5b-9ffe-4367-9e24-18a48af5181c-workload-certs\") pod \"45fa9c5b-9ffe-4367-9e24-18a48af5181c\" (UID: \"45fa9c5b-9ffe-4367-9e24-18a48af5181c\") "
Jul 15 21:38:16.054241 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:16.053785    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45fa9c5b-9ffe-4367-9e24-18a48af5181c-credential-socket\") pod \"45fa9c5b-9ffe-4367-9e24-18a48af5181c\" (UID: \"45fa9c5b-9ffe-4367-9e24-18a48af5181c\") "
Jul 15 21:38:16.054241 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:16.053812    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45fa9c5b-9ffe-4367-9e24-18a48af5181c-istio-data\") pod \"45fa9c5b-9ffe-4367-9e24-18a48af5181c\" (UID: \"45fa9c5b-9ffe-4367-9e24-18a48af5181c\") "
Jul 15 21:38:16.054241 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:16.053832    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxk56\" (UniqueName: \"kubernetes.io/projected/45fa9c5b-9ffe-4367-9e24-18a48af5181c-kube-api-access-lxk56\") pod \"45fa9c5b-9ffe-4367-9e24-18a48af5181c\" (UID: \"45fa9c5b-9ffe-4367-9e24-18a48af5181c\") "
Jul 15 21:38:16.054241 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:16.053877    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45fa9c5b-9ffe-4367-9e24-18a48af5181c-istio-token\") pod \"45fa9c5b-9ffe-4367-9e24-18a48af5181c\" (UID: \"45fa9c5b-9ffe-4367-9e24-18a48af5181c\") "
Jul 15 21:38:16.054241 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:16.053936    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45fa9c5b-9ffe-4367-9e24-18a48af5181c-istio-podinfo\") pod \"45fa9c5b-9ffe-4367-9e24-18a48af5181c\" (UID: \"45fa9c5b-9ffe-4367-9e24-18a48af5181c\") "
Jul 15 21:38:16.054241 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:16.053997    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45fa9c5b-9ffe-4367-9e24-18a48af5181c-workload-socket\") pod \"45fa9c5b-9ffe-4367-9e24-18a48af5181c\" (UID: \"45fa9c5b-9ffe-4367-9e24-18a48af5181c\") "
Jul 15 21:38:16.054241 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:16.054214    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45fa9c5b-9ffe-4367-9e24-18a48af5181c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "45fa9c5b-9ffe-4367-9e24-18a48af5181c" (UID: "45fa9c5b-9ffe-4367-9e24-18a48af5181c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:38:16.054624 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:16.054251    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45fa9c5b-9ffe-4367-9e24-18a48af5181c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "45fa9c5b-9ffe-4367-9e24-18a48af5181c" (UID: "45fa9c5b-9ffe-4367-9e24-18a48af5181c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 21:38:16.054624 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:16.054345    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45fa9c5b-9ffe-4367-9e24-18a48af5181c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "45fa9c5b-9ffe-4367-9e24-18a48af5181c" (UID: "45fa9c5b-9ffe-4367-9e24-18a48af5181c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:38:16.054624 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:16.054449    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45fa9c5b-9ffe-4367-9e24-18a48af5181c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "45fa9c5b-9ffe-4367-9e24-18a48af5181c" (UID: "45fa9c5b-9ffe-4367-9e24-18a48af5181c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:38:16.054783 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:16.054729    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45fa9c5b-9ffe-4367-9e24-18a48af5181c-istio-data" (OuterVolumeSpecName: "istio-data") pod "45fa9c5b-9ffe-4367-9e24-18a48af5181c" (UID: "45fa9c5b-9ffe-4367-9e24-18a48af5181c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:38:16.055936 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:16.055912    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45fa9c5b-9ffe-4367-9e24-18a48af5181c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "45fa9c5b-9ffe-4367-9e24-18a48af5181c" (UID: "45fa9c5b-9ffe-4367-9e24-18a48af5181c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 21:38:16.056040 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:16.056025    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45fa9c5b-9ffe-4367-9e24-18a48af5181c-istio-token" (OuterVolumeSpecName: "istio-token") pod "45fa9c5b-9ffe-4367-9e24-18a48af5181c" (UID: "45fa9c5b-9ffe-4367-9e24-18a48af5181c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 21:38:16.056404 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:16.056381    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45fa9c5b-9ffe-4367-9e24-18a48af5181c-kube-api-access-lxk56" (OuterVolumeSpecName: "kube-api-access-lxk56") pod "45fa9c5b-9ffe-4367-9e24-18a48af5181c" (UID: "45fa9c5b-9ffe-4367-9e24-18a48af5181c"). InnerVolumeSpecName "kube-api-access-lxk56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 21:38:16.056457 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:16.056440    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45fa9c5b-9ffe-4367-9e24-18a48af5181c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "45fa9c5b-9ffe-4367-9e24-18a48af5181c" (UID: "45fa9c5b-9ffe-4367-9e24-18a48af5181c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 15 21:38:16.155269 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:16.155238    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45fa9c5b-9ffe-4367-9e24-18a48af5181c-istio-envoy\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:38:16.155269 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:16.155266    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45fa9c5b-9ffe-4367-9e24-18a48af5181c-istiod-ca-cert\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:38:16.155269 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:16.155276    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45fa9c5b-9ffe-4367-9e24-18a48af5181c-workload-certs\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:38:16.155478 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:16.155284    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45fa9c5b-9ffe-4367-9e24-18a48af5181c-credential-socket\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:38:16.155478 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:16.155294    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45fa9c5b-9ffe-4367-9e24-18a48af5181c-istio-data\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:38:16.155478 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:16.155302    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxk56\" (UniqueName: \"kubernetes.io/projected/45fa9c5b-9ffe-4367-9e24-18a48af5181c-kube-api-access-lxk56\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:38:16.155478 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:16.155311    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45fa9c5b-9ffe-4367-9e24-18a48af5181c-istio-token\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:38:16.155478 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:16.155319    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45fa9c5b-9ffe-4367-9e24-18a48af5181c-istio-podinfo\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:38:16.155478 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:16.155326    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45fa9c5b-9ffe-4367-9e24-18a48af5181c-workload-socket\") on node \"ip-10-0-137-234.ec2.internal\" DevicePath \"\""
Jul 15 21:38:16.874671 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:16.874630    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-kcwtz_f4a0582f-65dd-47c9-91a1-e12bee5fcc56/manager/0.log"
Jul 15 21:38:16.906027 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:16.906001    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt"
Jul 15 21:38:16.906491 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:16.906002    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt" event={"ID":"45fa9c5b-9ffe-4367-9e24-18a48af5181c","Type":"ContainerDied","Data":"7cb58f865b5c9bcdfcd4b1662e86fe2f46dfecd088a40e1254ecc6ff6beb50ce"}
Jul 15 21:38:16.906491 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:16.906145    2566 scope.go:117] "RemoveContainer" containerID="eb48e463a778b1f2f46bf9f565aa48e5e88bd94ac96bafe1fcbba529de9faf9f"
Jul 15 21:38:16.929593 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:16.929567    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt"]
Jul 15 21:38:16.933627 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:16.933603    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-266bf8-openshift-default-7fb9bb959-9t8lt"]
Jul 15 21:38:16.990264 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:16.990236    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-xlrk4_4bc02df2-6841-4cda-b064-a19ea5f10523/manager/0.log"
Jul 15 21:38:17.129911 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:17.129811    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-567446f9b-dgm4c_73b59907-4d42-4445-b7b9-91a80790a8ba/manager/0.log"
Jul 15 21:38:17.638052 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:17.638026    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b849856cb-lcvdt_97177182-de80-4d9c-94ff-23c3300d11ce/copy-manifests/0.log"
Jul 15 21:38:17.651936 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:17.651911    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b849856cb-lcvdt_97177182-de80-4d9c-94ff-23c3300d11ce/manager/0.log"
Jul 15 21:38:17.756762 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:17.756732    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45fa9c5b-9ffe-4367-9e24-18a48af5181c" path="/var/lib/kubelet/pods/45fa9c5b-9ffe-4367-9e24-18a48af5181c/volumes"
Jul 15 21:38:18.294081 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:18.294051    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29735820-gm549_62fd87a2-6d1e-4bac-8354-862921c2587a/cleanup/0.log"
Jul 15 21:38:18.411617 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:18.411585    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29735835-x4vms_f021cabb-31c6-49c5-8fee-9574df40e4a0/cleanup/0.log"
Jul 15 21:38:18.664668 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:18.664583    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-8143c542-cxpf6_e1b4397d-0ffa-4c58-af9d-5dd79f8c79c1/revoke-keys/0.log"
Jul 15 21:38:18.782078 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:18.782050    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-af79a2b5-qvwzc_4ef064e5-f395-44e5-a27e-788ab15459b1/revoke-keys/0.log"
Jul 15 21:38:19.133306 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:19.133282    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-derive-6e3236bd-t7qlb_eb1054f6-f436-4f1b-a556-a6d4c394a44f/revoke-keys/0.log"
Jul 15 21:38:19.484413 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:19.484384    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-migrate-baea565a-jbcqp_ab0b03e8-299b-4e2a-a227-ba1d011fac09/revoke-keys/0.log"
Jul 15 21:38:19.604040 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:19.604009    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-61280292-tphkg_92c9974e-5ce2-40fe-aa2b-15994743a444/revoke-keys/0.log"
Jul 15 21:38:19.729805 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:19.729752    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-a3485fb9-rg8jc_a4be6efe-444a-47b9-96d2-4896b8f91e82/revoke-keys/0.log"
Jul 15 21:38:21.513586 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:21.513549    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-bdvjc_e5286d79-383d-4b78-a2b3-b8b6fea1b534/manager/0.log"
Jul 15 21:38:21.861468 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:21.861388    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-24hbb_f75b800f-3428-4734-9e43-98fedc98e8f5/registry-server/0.log"
Jul 15 21:38:21.986867 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:21.986835    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-l7nv5_856d10c0-5cc3-4e42-a3c1-9274c31e5e01/manager/0.log"
Jul 15 21:38:23.196601 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:23.196571    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-thxk8_ce43b1a9-5e27-4ac1-8430-644dd0d9b106/payload-pre-processing/0.log"
Jul 15 21:38:23.776829 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:23.776802    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-zb9gv_b0d97cd6-8f3d-4713-80e2-43d8be42fc6c/main/0.log"
Jul 15 21:38:23.887713 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:23.887687    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-xfcxw_b25440fc-4984-4e23-a0c0-dbf490e26002/main/0.log"
Jul 15 21:38:29.833370 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:29.833333    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:38:29.932262 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:29.932233    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log" limit=1024
Jul 15 21:38:32.961308 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:32.961278    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zzw8h_b9d9267e-0c23-42b2-bb48-55ed14050fff/global-pull-secret-syncer/0.log"
Jul 15 21:38:33.012429 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:33.012400    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6j89c_ad23d165-aeec-409f-99bd-12c27029221b/konnectivity-agent/0.log"
Jul 15 21:38:33.120802 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:33.120774    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-234.ec2.internal_4a1659746931c9e82b46a4829b7de6fa/haproxy/0.log"
Jul 15 21:38:38.573775 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:38.573743    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-bdvjc_e5286d79-383d-4b78-a2b3-b8b6fea1b534/manager/0.log"
Jul 15 21:38:38.701169 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:38.701138    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-24hbb_f75b800f-3428-4734-9e43-98fedc98e8f5/registry-server/0.log"
Jul 15 21:38:38.808267 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:38.808237    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-l7nv5_856d10c0-5cc3-4e42-a3c1-9274c31e5e01/manager/0.log"
Jul 15 21:38:41.575435 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:41.575403    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tjq5h_84bc5eba-a859-45de-8e04-2a65286f40f9/node-exporter/0.log"
Jul 15 21:38:41.622374 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:41.622345    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tjq5h_84bc5eba-a859-45de-8e04-2a65286f40f9/kube-rbac-proxy/0.log"
Jul 15 21:38:41.654442 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:41.654404    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tjq5h_84bc5eba-a859-45de-8e04-2a65286f40f9/init-textfile/0.log"
Jul 15 21:38:41.791194 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:41.791161    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f6164d2a-4adb-4ef7-8443-d53e915eb70e/prometheus/0.log"
Jul 15 21:38:41.811769 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:41.811746    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f6164d2a-4adb-4ef7-8443-d53e915eb70e/config-reloader/0.log"
Jul 15 21:38:41.843635 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:41.843569    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f6164d2a-4adb-4ef7-8443-d53e915eb70e/thanos-sidecar/0.log"
Jul 15 21:38:41.869787 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:41.869763    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f6164d2a-4adb-4ef7-8443-d53e915eb70e/kube-rbac-proxy-web/0.log"
Jul 15 21:38:41.896345 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:41.896325    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f6164d2a-4adb-4ef7-8443-d53e915eb70e/kube-rbac-proxy/0.log"
Jul 15 21:38:41.923664 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:41.923637    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f6164d2a-4adb-4ef7-8443-d53e915eb70e/kube-rbac-proxy-thanos/0.log"
Jul 15 21:38:41.949290 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:41.949270    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f6164d2a-4adb-4ef7-8443-d53e915eb70e/init-config-reloader/0.log"
Jul 15 21:38:42.771503 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:42.771471    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jhrn6/perf-node-gather-daemonset-l4z7z"]
Jul 15 21:38:42.771920 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:42.771833    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45fa9c5b-9ffe-4367-9e24-18a48af5181c" containerName="istio-proxy"
Jul 15 21:38:42.771920 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:42.771844    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="45fa9c5b-9ffe-4367-9e24-18a48af5181c" containerName="istio-proxy"
Jul 15 21:38:42.771920 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:42.771853    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a48bab9d-061c-45b7-8b8a-8e343d64058a" containerName="istio-proxy"
Jul 15 21:38:42.771920 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:42.771859    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a48bab9d-061c-45b7-8b8a-8e343d64058a" containerName="istio-proxy"
Jul 15 21:38:42.771920 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:42.771867    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd" containerName="maas-api"
Jul 15 21:38:42.771920 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:42.771872    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd" containerName="maas-api"
Jul 15 21:38:42.771920 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:42.771883    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f" containerName="maas-api"
Jul 15 21:38:42.771920 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:42.771889    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f" containerName="maas-api"
Jul 15 21:38:42.771920 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:42.771910    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a48bab9d-061c-45b7-8b8a-8e343d64058a" containerName="istio-proxy"
Jul 15 21:38:42.771920 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:42.771915    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a48bab9d-061c-45b7-8b8a-8e343d64058a" containerName="istio-proxy"
Jul 15 21:38:42.772246 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:42.771974    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="45fa9c5b-9ffe-4367-9e24-18a48af5181c" containerName="istio-proxy"
Jul 15 21:38:42.772246 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:42.771981    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a48bab9d-061c-45b7-8b8a-8e343d64058a" containerName="istio-proxy"
Jul 15 21:38:42.772246 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:42.771987    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a48bab9d-061c-45b7-8b8a-8e343d64058a" containerName="istio-proxy"
Jul 15 21:38:42.772246 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:42.771993    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0ee92b9-f3af-4ebe-8603-70ca3c3e43fd" containerName="maas-api"
Jul 15 21:38:42.772246 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:42.771999    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="45fa9c5b-9ffe-4367-9e24-18a48af5181c" containerName="istio-proxy"
Jul 15 21:38:42.772246 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:42.772005    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f4803f2-25e5-43aa-819b-6e5f6b3c7d6f" containerName="maas-api"
Jul 15 21:38:42.774881 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:42.774858    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jhrn6/perf-node-gather-daemonset-l4z7z"
Jul 15 21:38:42.778147 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:42.778123    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jhrn6\"/\"default-dockercfg-h4vxg\""
Jul 15 21:38:42.778441 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:42.778424    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jhrn6\"/\"openshift-service-ca.crt\""
Jul 15 21:38:42.779484 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:42.779464    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jhrn6\"/\"kube-root-ca.crt\""
Jul 15 21:38:42.785170 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:42.785146    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jhrn6/perf-node-gather-daemonset-l4z7z"]
Jul 15 21:38:42.885746 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:42.885715    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f4c1c280-b020-4c21-8714-3056dd560efb-proc\") pod \"perf-node-gather-daemonset-l4z7z\" (UID: \"f4c1c280-b020-4c21-8714-3056dd560efb\") " pod="openshift-must-gather-jhrn6/perf-node-gather-daemonset-l4z7z"
Jul 15 21:38:42.885961 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:42.885754    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f4c1c280-b020-4c21-8714-3056dd560efb-podres\") pod \"perf-node-gather-daemonset-l4z7z\" (UID: \"f4c1c280-b020-4c21-8714-3056dd560efb\") " pod="openshift-must-gather-jhrn6/perf-node-gather-daemonset-l4z7z"
Jul 15 21:38:42.885961 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:42.885787    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f4c1c280-b020-4c21-8714-3056dd560efb-lib-modules\") pod \"perf-node-gather-daemonset-l4z7z\" (UID: \"f4c1c280-b020-4c21-8714-3056dd560efb\") " pod="openshift-must-gather-jhrn6/perf-node-gather-daemonset-l4z7z"
Jul 15 21:38:42.885961 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:42.885863    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zc89\" (UniqueName: \"kubernetes.io/projected/f4c1c280-b020-4c21-8714-3056dd560efb-kube-api-access-8zc89\") pod \"perf-node-gather-daemonset-l4z7z\" (UID: \"f4c1c280-b020-4c21-8714-3056dd560efb\") " pod="openshift-must-gather-jhrn6/perf-node-gather-daemonset-l4z7z"
Jul 15 21:38:42.886081 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:42.885972    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f4c1c280-b020-4c21-8714-3056dd560efb-sys\") pod \"perf-node-gather-daemonset-l4z7z\" (UID: \"f4c1c280-b020-4c21-8714-3056dd560efb\") " pod="openshift-must-gather-jhrn6/perf-node-gather-daemonset-l4z7z"
Jul 15 21:38:42.987188 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:42.987146    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f4c1c280-b020-4c21-8714-3056dd560efb-proc\") pod \"perf-node-gather-daemonset-l4z7z\" (UID: \"f4c1c280-b020-4c21-8714-3056dd560efb\") " pod="openshift-must-gather-jhrn6/perf-node-gather-daemonset-l4z7z"
Jul 15 21:38:42.987188 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:42.987193    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f4c1c280-b020-4c21-8714-3056dd560efb-podres\") pod \"perf-node-gather-daemonset-l4z7z\" (UID: \"f4c1c280-b020-4c21-8714-3056dd560efb\") " pod="openshift-must-gather-jhrn6/perf-node-gather-daemonset-l4z7z"
Jul 15 21:38:42.987420 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:42.987233    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f4c1c280-b020-4c21-8714-3056dd560efb-lib-modules\") pod \"perf-node-gather-daemonset-l4z7z\" (UID: \"f4c1c280-b020-4c21-8714-3056dd560efb\") " pod="openshift-must-gather-jhrn6/perf-node-gather-daemonset-l4z7z"
Jul 15 21:38:42.987420 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:42.987272    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zc89\" (UniqueName: \"kubernetes.io/projected/f4c1c280-b020-4c21-8714-3056dd560efb-kube-api-access-8zc89\") pod \"perf-node-gather-daemonset-l4z7z\" (UID: \"f4c1c280-b020-4c21-8714-3056dd560efb\") " pod="openshift-must-gather-jhrn6/perf-node-gather-daemonset-l4z7z"
Jul 15 21:38:42.987420 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:42.987284    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f4c1c280-b020-4c21-8714-3056dd560efb-proc\") pod \"perf-node-gather-daemonset-l4z7z\" (UID: \"f4c1c280-b020-4c21-8714-3056dd560efb\") " pod="openshift-must-gather-jhrn6/perf-node-gather-daemonset-l4z7z"
Jul 15 21:38:42.987420 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:42.987351    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f4c1c280-b020-4c21-8714-3056dd560efb-sys\") pod \"perf-node-gather-daemonset-l4z7z\" (UID: \"f4c1c280-b020-4c21-8714-3056dd560efb\") " pod="openshift-must-gather-jhrn6/perf-node-gather-daemonset-l4z7z"
Jul 15 21:38:42.987420 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:42.987395    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f4c1c280-b020-4c21-8714-3056dd560efb-podres\") pod \"perf-node-gather-daemonset-l4z7z\" (UID: \"f4c1c280-b020-4c21-8714-3056dd560efb\") " pod="openshift-must-gather-jhrn6/perf-node-gather-daemonset-l4z7z"
Jul 15 21:38:42.987609 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:42.987425    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f4c1c280-b020-4c21-8714-3056dd560efb-lib-modules\") pod \"perf-node-gather-daemonset-l4z7z\" (UID: \"f4c1c280-b020-4c21-8714-3056dd560efb\") " pod="openshift-must-gather-jhrn6/perf-node-gather-daemonset-l4z7z"
Jul 15 21:38:42.987609 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:42.987453    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f4c1c280-b020-4c21-8714-3056dd560efb-sys\") pod \"perf-node-gather-daemonset-l4z7z\" (UID: \"f4c1c280-b020-4c21-8714-3056dd560efb\") " pod="openshift-must-gather-jhrn6/perf-node-gather-daemonset-l4z7z"
Jul 15 21:38:42.997034 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:42.997013    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zc89\" (UniqueName: \"kubernetes.io/projected/f4c1c280-b020-4c21-8714-3056dd560efb-kube-api-access-8zc89\") pod \"perf-node-gather-daemonset-l4z7z\" (UID: \"f4c1c280-b020-4c21-8714-3056dd560efb\") " pod="openshift-must-gather-jhrn6/perf-node-gather-daemonset-l4z7z"
Jul 15 21:38:43.085345 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:43.085253    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jhrn6/perf-node-gather-daemonset-l4z7z"
Jul 15 21:38:43.215265 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:43.215235    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jhrn6/perf-node-gather-daemonset-l4z7z"]
Jul 15 21:38:43.217248 ip-10-0-137-234 kubenswrapper[2566]: W0715 21:38:43.217219    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf4c1c280_b020_4c21_8714_3056dd560efb.slice/crio-bca38f0dabfdf16e39a63c51713a82553bb34c29c339c6205cd8ef38b8f2dcb7 WatchSource:0}: Error finding container bca38f0dabfdf16e39a63c51713a82553bb34c29c339c6205cd8ef38b8f2dcb7: Status 404 returned error can't find the container with id bca38f0dabfdf16e39a63c51713a82553bb34c29c339c6205cd8ef38b8f2dcb7
Jul 15 21:38:44.040054 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:44.040019    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jhrn6/perf-node-gather-daemonset-l4z7z" event={"ID":"f4c1c280-b020-4c21-8714-3056dd560efb","Type":"ContainerStarted","Data":"70d56dae7b2b1ad314a4ed93b62ba738b184292cecee9a8af4d5a4e0a744a793"}
Jul 15 21:38:44.040438 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:44.040074    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jhrn6/perf-node-gather-daemonset-l4z7z" event={"ID":"f4c1c280-b020-4c21-8714-3056dd560efb","Type":"ContainerStarted","Data":"bca38f0dabfdf16e39a63c51713a82553bb34c29c339c6205cd8ef38b8f2dcb7"}
Jul 15 21:38:44.040438 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:44.040103    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jhrn6/perf-node-gather-daemonset-l4z7z"
Jul 15 21:38:44.060054 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:44.060005    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jhrn6/perf-node-gather-daemonset-l4z7z" podStartSLOduration=2.059990795 podStartE2EDuration="2.059990795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 21:38:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:38:44.059036451 +0000 UTC m=+3057.208074509" watchObservedRunningTime="2026-07-15 21:38:44.059990795 +0000 UTC m=+3057.209028851"
Jul 15 21:38:46.396079 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:46.396047    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lmsms_7f53f2f8-84d0-4ee8-867c-f4735128a587/dns/0.log"
Jul 15 21:38:46.419108 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:46.419085    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lmsms_7f53f2f8-84d0-4ee8-867c-f4735128a587/kube-rbac-proxy/0.log"
Jul 15 21:38:46.615996 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:46.615967    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qvvx2_147d4538-e5f1-47af-b61c-f07de8e0b0df/dns-node-resolver/0.log"
Jul 15 21:38:47.284738 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:47.284692    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wcchl_1bfac382-72fc-400b-ab8b-9e4f53ab1d50/node-ca/0.log"
Jul 15 21:38:48.585411 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:48.585323    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-thxk8_ce43b1a9-5e27-4ac1-8430-644dd0d9b106/payload-pre-processing/0.log"
Jul 15 21:38:49.385450 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:49.385419    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-t2nhn_ecc83405-3c34-4022-be1c-be802104e8b6/serve-healthcheck-canary/0.log"
Jul 15 21:38:50.055079 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:50.055051    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jhrn6/perf-node-gather-daemonset-l4z7z"
Jul 15 21:38:50.080262 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:50.080231    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kwmkr_075a9740-f176-4bb6-9d0a-9db3ba48fc75/kube-rbac-proxy/0.log"
Jul 15 21:38:50.110808 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:50.110781    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kwmkr_075a9740-f176-4bb6-9d0a-9db3ba48fc75/exporter/0.log"
Jul 15 21:38:50.142076 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:50.142045    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kwmkr_075a9740-f176-4bb6-9d0a-9db3ba48fc75/extractor/0.log"
Jul 15 21:38:52.845838 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:52.845807    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-kcwtz_f4a0582f-65dd-47c9-91a1-e12bee5fcc56/manager/0.log"
Jul 15 21:38:52.994967 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:52.994932    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-xlrk4_4bc02df2-6841-4cda-b064-a19ea5f10523/manager/0.log"
Jul 15 21:38:53.090141 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:53.090112    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-567446f9b-dgm4c_73b59907-4d42-4445-b7b9-91a80790a8ba/manager/0.log"
Jul 15 21:38:53.323753 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:53.323718    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b849856cb-lcvdt_97177182-de80-4d9c-94ff-23c3300d11ce/manager/0.log"
Jul 15 21:38:53.342781 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:53.342757    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b849856cb-lcvdt_97177182-de80-4d9c-94ff-23c3300d11ce/copy-manifests/0.log"
Jul 15 21:38:54.092321 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:54.092279    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-thxk8_ce43b1a9-5e27-4ac1-8430-644dd0d9b106/payload-pre-processing/0.log"
Jul 15 21:38:54.742052 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:54.741965    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6bd6785cdc-s9bmx_5557b001-8b97-4d1e-a501-fdb11e14c1c4/manager/0.log"
Jul 15 21:38:55.608803 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:55.608773    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-thxk8_ce43b1a9-5e27-4ac1-8430-644dd0d9b106/payload-pre-processing/0.log"
Jul 15 21:38:59.897654 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:59.897618    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-s9gwp_6e3d4282-65a0-4a65-90c1-1da3cf585ff4/migrator/0.log"
Jul 15 21:38:59.920105 ip-10-0-137-234 kubenswrapper[2566]: I0715 21:38:59.920068    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-s9gwp_6e3d4282-65a0-4a65-90c1-1da3cf585ff4/graceful-termination/0.log"